Nvidia H100 GPUs: Supply and Demand

The real gut-punch for this is a reminder how far behind most engineers are in this race. With web 1.0 and web 2.0 at least you could rent a cheap VPS for $10&#x2F;month and try out some stuff. There is almost no universe where a couple of guys in their garage are getting access to 1000+ H100s with a capital cost in the multiple millions. Even renting at that scale is $4k&#x2F;hour. That is going to add up quickly.<p>I hope we find a path to at least fine-tuning medium sized models for prices that aren&#x27;t outrageous. Even the tiny corp&#x27;s tinybox [1] is $15k and I don&#x27;t know how much actual work one could get done on it.<p>If the majority of startups are just &quot;wrappers around OpenAI (et al.)&quot; the reason is pretty obvious.<p>1. <a href="https:&#x2F;&#x2F;tinygrad.org&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;tinygrad.org&#x2F;</a>

&gt;Who is going to take the risk of deplying 10,000 AMD GPUs or 10,000 random startup silicon chips? That’s almost a $300 million investment.<p>Ironically, Jensen Huang did something like this many years ago. In an interview for his alma mater, he tells the story about how he had bet the existence of Nvidia on the successful usage of a new circuit simulation computer from a random startup that allowed Nvidia to complete the design of their chip.

What nobody is talking about here is that there is no more power available in the US. All the FAANGS have scooped up the space and power contracts.<p>You can buy all the GPUs you can possibly find. If you want to deploy 10MW+, it just doesn&#x27;t exist.<p>These things need redundant power&#x2F;cooling, real data centers, and can&#x27;t just be put into chicken farms. Anything less than 10MW isn&#x27;t enough compute now either for large scale training and you can&#x27;t spread it across data centers because all the data needs to be in one place.<p>So yea... good luck.

Its really time for some competition. Either AMD or some chinese company like &#x27;more threads&#x27; need to speed up and get something on the market to break the Nvidia dominance. Nvidia is showing already some nasty typical evil behavior that has to be stopped. I know not easy with fully booked partners at Samsung&#x2F;tmct&#x2F;etc

It&#x27;s weird not more is made of the fact the Google&#x27;s TPUs aren&#x27;t the only real, shipping, credible alternative to NVidia.<p>I wonder how much a TPU company would be worth if Google spun it off and it started selling them?

Having worked with Quants before, the reality is that however big your compute farm, they will want more. I think this is what is going on with these large AI companies - they are simply utilising all of the resource they have.<p>Of course they could do with more GPUs. If you gave them 1,000x their current number, they&#x27;d think up ways of utilising all of them, and have the same demand for more. This is how it should be.

I think the author has missed a pretty large segment of demand. Non-cloud&#x2F;non-tech enterprises are also buying large quantities of H100s and A100s for their own machine learning and simulation workloads. Where I work, we are going to have more than 1000 H100s by the end of the year, I am very excited to start benchmarking them soon :)

It&#x27;s weird that this article ignores the entire traditional HPC market&#x2F;DoE&#x2F;DoD demand for H100s.

If the bottleneck isn’t TSMC wafer starts but CoWoS, where exactly does that bottleneck come from? From what I understand, its the interposer connecting GPU and HBM wafers. Are they hard to make, is the yield bad, are there insufficient production lines, …?

Is Nvidia even able to capture a proportionately significant amount of revenue from increases in demand for GPU cycles? As the article describes, there are real bottlenecks, but how does this play out? My assumption is that Nvidia doesn&#x27;t have proportional pricing power for some reason. If demand increases 10x, they can&#x27;t raise prices to the same extent (correct me if I&#x27;m wrong).<p>How would that even play out then? Is everyone in the world simply stuck waiting for Nvidia&#x27;s capacity to meet demand?<p>There is obviously a huge incentive now to be competitive here, but is it realistic that anyone else might meaningfully meet demand before Nvidia can?

Very good article ! Nice insight as to who&#x2F;what&#x2F;where and how much :)

By far the biggest issue it utilisation of the GPUs, if people worked on that instead of throwing more power at problems this would be way less of a problem

The discussion on how different industries are vying for the same limited supply of components adds another layer of complexity to the situation. It&#x27;s intriguing to see how GPUs are being sought after by diverse sectors, and this phenomenon showcases the versatility and importance of these components in modern technology.

This is a very high quality writeup.

Jenson could write one of the clouds a license to use 4090s in a DC and make this crunch disappear overnight (would be rough for gamers though)

I know it’s off topic, forgive me HN Gods, but this was right at the top of the article and threw me off:<p>&gt; Elon Musk says that “GPUs are at this point considerably harder to get than drugs.”<p>Does Elon have a hard time getting drugs?

What is with this webpage, it appears to be blocked by pretty much every browser?

This is a pretty awful article.<p>AFAICT it consists of a bunch of anecdotes by thought-leader types followed by a corny-ass <i>song</i>.<p>HN, you can do better. I believe in you. Try harder.