Matrix-vector multiplication implemented in off-the-shelf DRAM for Low-Bit LLMs

Some more background information:<p>One of the original proposals for in-DRAM compute: <a href="https:&#x2F;&#x2F;users.ece.cmu.edu&#x2F;~omutlu&#x2F;pub&#x2F;in-DRAM-bulk-AND-OR-ieee_cal15.pdf" rel="nofollow">https:&#x2F;&#x2F;users.ece.cmu.edu&#x2F;~omutlu&#x2F;pub&#x2F;in-DRAM-bulk-AND-OR-ie...</a><p>First demonstration with off-the-shelf parts: <a href="https:&#x2F;&#x2F;parallel.princeton.edu&#x2F;papers&#x2F;micro19-gao.pdf" rel="nofollow">https:&#x2F;&#x2F;parallel.princeton.edu&#x2F;papers&#x2F;micro19-gao.pdf</a><p>DRAM Bender, the tool they are using to implement this: <a href="https:&#x2F;&#x2F;github.com&#x2F;CMU-SAFARI&#x2F;DRAM-Bender">https:&#x2F;&#x2F;github.com&#x2F;CMU-SAFARI&#x2F;DRAM-Bender</a><p>Memory-Centric Computing: Recent Advances in Processing-in-DRAM<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2412.19275" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2412.19275</a>

Did anyone else notice the absolutely insane author lists of references 1 and 3?<p>I was expecting to find this 2016 article in there: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=12469270">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=12469270</a><p>This 2019 one does show up: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=22712811">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=22712811</a><p>Of course, this &quot;out of spec&quot; behaviour of DRAM, more specifically the ability to do copying, is also implicated in this infamous bug: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=5314959">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=5314959</a><p>It seems more than one person independently observed such a thing, and thought &quot;this might be a useful behaviour&quot;.

<i>&gt; By intentionally issuing DRAM commands that violate manufacturer-specified timing parameters.. [gaining] massive parallelism up to 65,536 bitwise operations in parallel.</i><p>Take that, binary blobs for DRAM training!

This is just mind-bendingly weird and wonderfully creative. It can pay to work in the weeds! Bravo.

They&#x27;re doing matrix operations in the Dram itself? That sounds insane and also fascinating

In the hardware world are there risks of taking advantage of a bug knowing that the manufacturer may someday fix the bug? I know in the software world it&#x27;s a bad idea to leverage a bug in a platform to enable a feature (or fix another bug). The bug you&#x27;re counting on being present may get fixed 15 years in the future and then your system explodes and no one knows why.<p>edit: seems like there was a recent discussion about something similar... undefined behavior in some C function iirc

&gt;General matrix-vector multiplication (GeMV)<p>Ok, so my math isnt great.<p>When I was studying Quaternions during my 3d math class (That I failed the first time, like I said, not a math guy) they briefly covered the history of matrix calculation in graphics development.<p>My understanding is that Quaternions became popular because they are <i>almost</i> as accurate as matrices but much less complex computationally.<p>Has anyone tried building an LLM using Quats instead of matrices?<p>Or are the optimisations with Quaternions more useful in realtime?

A bit unscientific that they don&#x27;t cite the original Intelligent RAM (IRAM) sources from 1997:<p><a href="https:&#x2F;&#x2F;scholar.google.com&#x2F;scholar?hl=en&amp;as_sdt=0%2C5&amp;q=iram+patterson&amp;btnG=" rel="nofollow">https:&#x2F;&#x2F;scholar.google.com&#x2F;scholar?hl=en&amp;as_sdt=0%2C5&amp;q=iram...</a>

Can we expect to see matrix multiplication and perhaps other ops move from classic CPUs out into the DRAM, perhaps with deliberate hardware support?<p>And does such a processing shift give advantage to Samsung etc? Where does this leave NVIDIA etc?

Funny hack. Without having read the paper I&#x27;d assume the operations to be thermally unstable. So LLM inference results will vary based on environmental temperature :-)

This woule be a cool way to make a cheap inferencing device for the largest LLMs

So is this a new technique of doing computations within existing DRAM to overcome the memory wall issue of modern computing?