Which GPU(s) to Get for Deep Learning

Evaluating AMD GPU by their specs is not going to paint the full picture. Their drivers are a serious problem. I&#x27;ve managed to get ROCm mostly working on my system (ignoring all the notifications of what is officially supported, the jammy debs from the official repo seem to work on Debian testing). The range of supported setups is limited so it is quite easy to end up in a similar situation.<p>I expect system lockups when doing any sort of model inference. From the experiences of the last few years I assume it is driver bugs. Based on their rate of improvement they probably will get there in around 2025, but their past performance has been so bad I wouldn&#x27;t recommend buying a card for machine learning until they&#x27;ve proven that they&#x27;re taking the situation seriously.<p>Although in my opinion buy AMD anyway if you need a GPU on linux. Their open source drivers are a lot less hassle as long as you don&#x27;t need BLAS.

Just as an FYI&#x2F;additional data point, I bought a 3090 FE from Ebay a few months ago for £605 including delivery.<p>I&#x27;ve only just started using it for Llama running locally on my computer at home and I have to say... colour me impressed.<p>It generates the output slightly faster than reading speed so for me it works perfectly well.<p>The 24GB of VRAM should keep it relevant for a bit too and I can always buy another and NVLink them should the need arise.

I used Tim&#x27;s guide to build a dual RTX 3090 PC, paying 2300€ in total by getting used components. It can run inference of Llama-65B 4bit quantized at more than 10tok&#x2F;s.<p>Specs: 2x RTX 3090, NVLink Bridge, 128GB DDR4 3200 RAM, Ryzen 7 3700X, X570 SLI mainboard, 2TB M.2 NVMe SSD, air cooled mesh case.<p>Finding the 3-slot nvlink bridge is hard and it&#x27;s usually expensive. I think it&#x27;s not worth it in most cases. I managed to find a cheap used one. Cooling is also a challenge. The cards are 2.7 slots wide and the spacing is usually 3 slots, so there isn&#x27;t much room. Some people are putting 3d printed shrouds on the back of the PC case to suck the air out of the cards with an extra external fan. Also limiting the power from 350W to 280W or so per card doesn&#x27;t cost a lot of performance. The CPU is not limiting the performance at all, as long as you have 4 cores per GPU you&#x27;re good.

I hear a lot about CUDA and how bad ROCm is etc. and I’ve been trying to understand what exactly CUDA is doing that is so special; isn’t the maths for neural networks mostly multiplying large arrays&#x2F;tensors together? What magic is CUDA doing that is so different for other vendors to implement? Is it just lock-in, the type of operations that are available, some kind of magical performance advantage or something else that CUDA is doing?

You can tell how NVIDIA dominants the market by the fact their price&#x2F;performance &quot;curve&quot; is almost a straight line.<p>In a competitive market that line has distortions where one player trts to undercut the other.<p>There are no bargains because there is almost no competitive pressure and so there is barely any distortion in that line.

So Nvidia is going to pretty much corner the market for a long time? This bit I expected but was still sad to read. Surely we would benefit from competition. It would probably take a lot of investment from AMD to make that happen, I imagine.<p>&gt; AMD GPUs are great in terms of pure silicon: Great FP16 performance, great memory bandwidth. However, their lack of Tensor Cores or the equivalent makes their deep learning performance poor compared to NVIDIA GPUs. Packed low-precision math does not cut it. Without this hardware feature, AMD GPUs will never be competitive.<p>Edit: what about Intel arc GPU? Any hope there?

App based on this post to help you decide what to buy: <a href="https:&#x2F;&#x2F;nanx.me&#x2F;gpu&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;nanx.me&#x2F;gpu&#x2F;</a>

I think there&#x27;s one more axis: Frequency-of-use.<p>For occasionally use, the major constraint isn&#x27;t speed so much as which models fit. I tend to look at $&#x2F;GB VRAM as my major spec. Something like a 3060 12GB is an outlier for fitting sensible models while being cheap.<p>I don&#x27;t mind waiting a minute instead of 15 seconds for some complex inference if I do it a few times per day. Or having training be slower if it comes up once every few months.

I&#x27;m sticking with nVidia for now (currently a 3090 bought secondhand of eBay) as it is the most tested&#x2F;supported by far, but it is great to see AMD making progress (finally) as some competition in this segment is desperatly needed.

I&#x27;ve bought a Radeon RX 6700XT (12GB) last year, primarily for playing games.<p>But after Stable Diffusion came out, I started to play around with it and was pleasantly surprised that the GPU could handle it!<p>The setup is a little messy, and Linux only.<p>For someone targeting AI, definitely pick an Nvidia card with 12+ GBs of VRAM.

You&#x27;ll want lots of memory, so depends on your price point.<p>4090 ($1,600) &gt; 3090 ($1300 new - $600 used) &gt; 3060 ($300)<p>used 3090 is ideal value. Lots of models will need the 24gb ram

Trying to build a scalable home 4090 cluster but running into a lot of confusion...<p>Let&#x27;s say<p>- I have a motherboard + cpu + other components and they&#x27;ve both got plenty of pcie lanes to spare, total this part draws 250W (incl the 25% extra wattage headroom)<p>- start off with one RTX 4090, TDP 450W, with headroom ~600W.<p>- I want to scale up by adding more 4090s over time, as many as my pcie lanes can support.<p><pre><code> 1. How do I add more PSUs over time? 2. Recommended initial PSU wattage? Recommended wattage for each additional pair of 4090s? 3. Recommended PSU brands and models for my use case? 4. Is it better to use PCI gen5 spec-rated PSUs? ATX 3.0? 12vhpwr cables rather than the ordinary 8-pin cables? I&#x27;ve also read somewhere that power cables between different brands of PSUs are *not* interchangeable?? 5. Whenever I add an additional PSU, do I need to do something special to electrically isolate the PCIe slots? 6. North American outlets are rated for ~15A * 120V. So roughly 1800W. I can just use one outlet per psu whenever it&#x27;s under 1800W, right? For simplicity let&#x27;s also ignore whatever load is on that particular electrical circuit. </code></pre> Each GPU means another 600W. Let&#x27;s say I want to add another PSU for every 2 4090s. I understand that to sync the bootup of multiple PSUs you need an add2psu adapter.<p>I understand the motherboard can provide ~75W for a pcie slot. I take it that the rest comes from the psu power cables. I&#x27;ve seen conflicting advice online - apparently miners use pcie x1 electrically isolated risers for additional power supplies, but also I&#x27;ve seen that it&#x27;s fine as long as every input power cable for 1 gpu just comes from one psu, regardless of whether it&#x27;s the one that powers the motherboard. Either way x1 risers is an unattractive option bc of bandwidth limitations.<p>pls help

One, don&#x27;t use a case. Look at how miners mounted their hardware on racks and take notes. Cheaper, better for temps, and the most efficient use of space.<p>Two, I recommend ignoring electricity cost and using all you can. If it&#x27;s cheaper now than it ever will be, use it while it&#x27;s cheap. If it will go down due to renewables, nuclear, etc in the future, it&#x27;s good to buy up the GPUs while their price is artificially depressed from energy fears.<p>Third, go for server type PSUs and breakout boards. The server PSUs cant be beaten in watts for your dollar, and are extremely efficient.<p>Finally, consider scooping up some x79 and x99 xeon boards from Chinese sellers. They&#x27;re cheap as hell, have PCI lanes out the wazoo, etc. This means you don&#x27;t have to fool with as many mobos to run the same amount of gpus. If you go this route, don&#x27;t get the bottom of the barrel no-name motherboards. Machinist is a decent one.

There’s clearly demand to buy AI capable GPUs at the store at a low price.<p>But Nvidias monopoly mean a they cripple their retail cards and push the AI stuff to data centers.<p>If only there was many manufacturers of AI hardware and software there would be abundant cheap products at every level.<p>AMD and Intel don’t seem to be able to compete and there’s no sign that will change.<p>So AI is going to remain expensive and hard to get for a very long time.

I almost immediately became suspicious on the accuracy of this article when they said the &quot;Nvidia RTX 40 Ampere series&quot;. Ampere was the architecture name for the RTX 30 series. Ada Lovelace is the architecture name for the RTX 40 series.

Any advice for mobile gpus? I&#x27;m interested in getting a laptop (preferably in the portable category). Obviously it&#x27;s not going to be in 4090 territory, that&#x27;s a tradeoff I&#x27;m willing to make.

Weird to leave out Apple. They seem to be the cheapest option to get a large amount of GPU memory.

4090 is now in high end PCs, with 24GB VRAM, that&#x27;s what I&#x27;m going to buy.<p>Everyone talks about Nvidia GPUs and AMD MI250&#x2F;MI300, where is Intel? Would love to have a 3rd player.

Raw performance rating for the RTX 3070 seems very weirdly placed in the chart. It&#x27;s below the RTX 3060 Ti, which doesn&#x27;t seem to make any sense.

I never tire of this. Tim is a wonderful no nonsense person. I love these posts and I love that it stays up to date.

Really a shame that the 4070ti doesn&#x27;t have 16GB.<p>But I guessed it is expected that Nvidia doesn&#x27;t want to cannibalize the 4080.

For a compromise, how is the recently released 4060ti with 16gb RAM? Its about a third the price of a 4090.

Do local GPUs make sense? For the same price, can&#x27;t you got a full years worth of cloud gpu time?

Omg that&#x27;s a long read but very informative