Llama.cpp AI Performance with the GeForce RTX 5090 Review

It used to be that for locally running GenAI, VRAM per dollar was king, so used NVidia RTX 3090 cards were the undisputed darlings of DYI LLM with 24GB for 600€-800€ or so. Sticking two of these in one PC isn&#x27;t too difficult despite them using 350W each.<p>Then Apple introduced Macs with 128 GB and more unified memory at 800GB&#x2F;s and the ability to load models as large as 70GB (70b FP8) or even larger ones. The M1 Ultra was unable to take full advantage of the excellent RAM speed, but with the M2 and the M3, performance is improving. Just be prepared to spend 5000€ or more for a M3 Ultra. Another alternative would be a EPYC 9005 system with 12x DDR5-6000 RAM for 576GB&#x2F;s of memory bandwidth with the LLM (preferably MoE) running on the CPU instead of a GPU.<p>However today, with the latest, surprisingly good reasoning models like QwQ-32B using up thousands or tens of thousands of tokens in their replies, performance is getting more important than previously and these systems (Macs and even RTX 3090s) might fall out of favor, because waiting for a finished reply will take several minutes or even tens of minutes. Nvidia Ampere and Apple silicon (AFAIK) are also missing FP4 support in hardware, which doesn&#x27;t help.<p>For the same reason AMD Halo Strix with a mere 273GB&#x2F;s of RAM bandwidth and perhaps also NVidia Project Digits (also speculated to offer similar RAM bandwidth) might just be too slow for reasoning models with more than 50GB or so of active parameters.<p>On the other hand, if the prices for the RTX 5090 remain at 3500€, they will likely remain insignificant for the DIY crowd for that reason alone.<p>Perhaps AMD will take the crown with a variant of their RDNA4 RX 9070 card with 32GB of VRAM priced at around 1000€? Probably wishful thinking…

The benchmark only touches 8B-class models at 8-bit quantification. Would be interesting to see how it fares with models that use more of the card ram, and under varying quantization and context lengths.

Sometimes I daydream about a world where GPUs just have the equivalent of LPCAMM and you could put in as much RAM as you can afford and as much as the hardware supports, much like is the case with motherboards, even if something along the way would bottleneck somewhat. It&#x27;d really extend the life of some hardware, yet companies don&#x27;t want that.<p>That said, it&#x27;s cool that you can even get an L4 with 24 GB of VRAM that actually performs okay, yet is <i>passively cooled</i> and consumes like 70W, at that point you can throw a bunch of them into a chassis and if you haven&#x27;t bankrupted yourself by then, they&#x27;re pretty good.<p>I did try them out on Scaleway, the pricing isn&#x27;t even that exorbitant, using consumer GPUs for LLM use cases doesn&#x27;t quite hit the same since.

Correct me if I&#x27;m wrong, but I have the impression that we&#x27;d usually expect to see bigger efficiency gains while these are marginal?<p>If so that would confirm the notion that they&#x27;ve hit a ceiling and pushing against physical limitations.

It would be very interesting to see these alongside benchmarks for Apple M4, AMD Halo Strix and other AMD cards.

If my budget is only ~$1000 should I buy a used 3090 or a new 5080? (for AI, I don&#x27;t care about gaming)

Curious if the author checked whether his card doesn&#x27;t have any missing ROPs.

P40s are so slept on. 24gb vram for $150.

96gb beast coming from nvidia.<p><a href="https:&#x2F;&#x2F;videocardz.com&#x2F;newz&#x2F;nvidia-rtx-pro-6000-blackwell-leaked-24064-cores-96gb-g7-memory-and-600w-double-flow-through-cooler" rel="nofollow">https:&#x2F;&#x2F;videocardz.com&#x2F;newz&#x2F;nvidia-rtx-pro-6000-blackwell-le...</a>

Is llama.cpp&#x27;s CUDA implementation decent? (e.g. does it use CUTLASS properly or something more low level)

Great to see Mr Larabel@Phoronix both maintaining consistent legit reporting and still have time for one-offs like this in these times of AI slop and other OG writers either quitting or succumbing to the vortex. Hats off!

And the latest AMD cards for reference?<p>Also, some DeepSeek models would be cool.

TL;DR; performance isn&#x27;t bad, but perf per Watt isn&#x27;t better than 4080 or 4090 and can even be significantly lower than 4090 in certain contexts.