Microsoft researchers developed a hyper-efficient AI model that can run on CPUs

Repo with demo video and benchmark:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;microsoft&#x2F;BitNet">https:&#x2F;&#x2F;github.com&#x2F;microsoft&#x2F;BitNet</a><p>&quot;...It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption...&quot;<p><a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2402.17764" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2402.17764</a>

This will happen more and more. This is why NVidia is rushing to get CUDA a software level lock-in otherwise their stock will go the way of Zoom.

&quot;Parameter count&quot; is the &quot;GHz&quot; of AI models: the number you&#x27;re most likely to see but least likely to need. All of the models compared (in the table on the huggingface link) are 1-2 billion parameters but the models range in actual size by more than a factor of 10.

I think almost all the free LLMs (not AI) that you find on hf can &#x27;run on CPUs&#x27;.<p>The claim here seems to be that it runs <i>usefully fast</i> on CPU.<p>We&#x27;re not sure how accurate this claim is, because we don&#x27;t know how fast this model runs on a GPU, because:<p><pre><code> &gt; Absent from the list of supported chips are GPUs [...] </code></pre> And TFA doesn&#x27;t really quantify anything, just offers:<p><pre><code> &gt; Perhaps more impressively, BitNet b1.58 2B4T is speedier than other models of its size — in some cases, twice the speed — while using a fraction of the memory. </code></pre> The model they link to is just over 1GB in size, and there&#x27;s plenty of existing 1-2GB models that are quite serviceable on even a mildly-modern CPU-only rig.

This is over a year old. The sky did not come down, everyone didn&#x27;t switch to this in spite of the &quot;advantages&quot;. If you look into why, you&#x27;ll see that it does, in fact, affect the metrics, and some more than others, and there is no silver bullet.

The pricing war will continue to rock bottom

Why do they call it &quot;1-bit&quot; if it uses ternary {-1, 0, 1}? Am I missing something?

There are projects working on distributed LLMs, such as exo[1]. If they can crack the distributed problem fully and get performance it’s a game changer. Instead of spending insane amounts on Nvidia GPUs, can just deploy commodity clusters of AMD EPYC servers with tons of memory, NVMe disks, and 40G or 100G networking which is vastly less expensive. Goodbye Nvidia AI moat.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;exo-explore&#x2F;exo">https:&#x2F;&#x2F;github.com&#x2F;exo-explore&#x2F;exo</a>

Super cool. Imagine specialized hardware for running these.

Is there a library to distill bigger models into BitNet?

&gt; it’s openly available under an MIT license and can run on CPUs, including Apple’s M2.<p>Weird comparison? The M2 already runs 7 or 13gb LLama and Mistral models with relative ease.<p>The M-series and Macbooks are so ubiquitous that perhaps we&#x27;re forgetting how weak the average CPU (think i3 or i5) can be.

..and eventually the Skynet Funding Bill was passed.