Nvidia reveals new A.I. chip, says costs of running LLMs will drop significantly

The GH200 was announced in May. What’s new is a variant with HBM3e memory which increases capacity (96GB -&gt; 144GB) and bandwidth (4TB&#x2F;s -&gt; 5TB&#x2F;s). The posted article butchers this. Here’s a much better source: <a href="https:&#x2F;&#x2F;www.anandtech.com&#x2F;show&#x2F;20001&#x2F;nvidia-unveils-gh200-grace-hopper-gpu-with-hbm3e-memory" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.anandtech.com&#x2F;show&#x2F;20001&#x2F;nvidia-unveils-gh200-gr...</a>

Although this is obviously great news, it is increasingly troubling how AI innovation (from the hardware sector) is pretty much limited to Nvidia.<p>I haven&#x27;t really gotten too much into it, so I&#x27;m not sure how has Nvidia come to absolutely dominate this market; AMD GPUs are quite good in the gaming sector... though I guess... with just two real players in the GPU market it&#x27;s difficult to really get anywhere.

Regarding NVIDIA domination, I want to promote that the Google project <a href="https:&#x2F;&#x2F;github.com&#x2F;openxla&#x2F;iree">https:&#x2F;&#x2F;github.com&#x2F;openxla&#x2F;iree</a> exists and IREE acts as a way to turn Tensorflow, Pytorch, and MLIR workflows to compute on cpu, vulkan compute, cuda, rocm, metal and others.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;RechieKho&#x2F;IREE.gd">https:&#x2F;&#x2F;github.com&#x2F;RechieKho&#x2F;IREE.gd</a> -- RechieKho and I collaborate on making this work for Godot Engine, but IREE.gd is at a proof of concept stage.

JM2C<p>lood_in_4bit=True will let you run Llama2-7B variants at 6.3GB VRAM.