100x defect tolerance: How we solved the yield problem

I think this is an important step, but it skips over that &#x27;fault tolerant routing architecture&#x27; means you&#x27;re spending die space on routes vs transistors. This is exactly analogous to using bits in your storage for error correcting vs storing data.<p>That said, I think they do a great job of exploiting this technique to create a &quot;larger&quot;[1] chip. And like storage it benefits from every core is the same and you don&#x27;t need to get to every core directly (pin limiting).<p>In the early 2000&#x27;s I was looking at a wafer scale startup that had the same idea but they were applying it to an FPGA architecture rather than a set of tensor units for LLMs. Nearly the exact same pitch, &quot;we don&#x27;t have to have all of our GLUs[2] work because the built in routing only uses the ones that are qualified.&quot; Xilinx was still aggressively suing people who put SERDES ports on FPGAs so they were pin limited overall but the idea is sound.<p>While I continue to believe that many people are going to collectively lose trillions of dollars ultimately pursuing &quot;AI&quot; at this stage. I appreciate the the amount of money people are willing to put at risk here allow for folks to try these &quot;out of the box&quot; kinds of ideas.<p>[1] It is physically more cores on a single die but the overall system is likely smaller, given the integration here.<p>[2] &quot;Generic Logic Unit&quot; which was kind of an extended LUT with some block RAM and register support.

So they massively reduce the area lost to defects per wafer, from 361 to 2.2 square mm. But from the figures in this blog, this is massively outweighed by the fact that they only get 46222 sq mm useable area out of the wafer, as opposed to 56247 that the H100 gets - because they are using a single square die instead of filling the circular wafer with smaller square dies, they lose 10,025 sq mm!<p>Not sure how that&#x27;s a win.<p>Unless the rest of the wafer is useable for some other customer?

Neat. What about power density?<p>An H100 has a TDP of 700 watts (for the SXM5 version). With a die size of 814 mm^2 that&#x27;s 0.86 W&#x2F;mm^2. If the cerebras chip has the same power density, that means a cerebras TDP of 37.8 kW.<p>That&#x27;s a lot. Let&#x27;s say you cover the whole die area of the chip with water 1 cm deep. How long would it take to boil the water starting from room temperature (20 degrees C)?<p>amount of water = (die area of 46225 mm^2) * (1 cm deep) * (density of water) = 462 grams<p>energy needed = (specific heat of water) * (80 kelvin difference) * (462 grams) = 154 kJ<p>time = 154 kJ &#x2F; 39.8 kW = 3.9 seconds<p>This thing will boil (!) a centimeter of water in 4 seconds. A typical consumer water cooler radiator would reduce the temperature of the coolant water by only 10-15 C relative to ambient, and wouldn&#x27;t like it (I presume) if you pass in boiling water. To use water cooling you&#x27;d need some extreme flow rate and a big rack of radiators, right? I don&#x27;t really know. I&#x27;m not even sure if that would work. How do you cool a chip at this power density?

To summarize: localize defect contamination to a very small unit size, by making the cores tiny and redundant.<p>Analogous to a conglomerate wrapping each business vertical in a limited liability veil so that lawsuits and bankruptcy do not bring down the whole company. The smaller the subsidiaries, the less defect contamination but also the less scope for frictionless resource and information sharing.

I live in a small city&#x2F;large town that has a large number of craft breweries. I always marveled at how these small operations were able to churn out so many different varieties. Turns out they are actually trying to make their few core recipes but the yield is so low they market the less consistent results as...all that variety I was so impressed with.

&gt; Second, a cluster of defects could overwhelm fault tolerant areas and disable the whole chip.<p>That’s an interesting point. In architecture class (which was basic and abstract so I’m sure Cerebras is doing something much more clever), we learned that defects cluster, but this is a good thing. A bunch of defects clustering on one core takes out the core, a bunch of defects not clustering could take out… a bunch of cores, maybe rendering the whole chip useless.<p>I wonder why they don’t like clustering. I could imagine in a network of little cores, maybe enough defects clustered on the network could… sort of overwhelm it, maybe?<p>Also I wonder how much they benefit from being on one giant wafer. It is definitely cool as hell. But could chiplets eat away at their advantage?

TSMC also have a manufacturing process used by Tesla&#x27;s Dojo where you can cut up the chips, throw away the defective ones, and then reassemble working ones into a sort of wafer scale device (5x5 chips for Dojo). Seems like a more logical design to me.

I assume people are aware, but Cerebras has a web demo and API which is open to try and it is 2000 tokens per second for Llama 3.3 70b and 1000 tokens per second for Llama 3.1 405b.<p><a href="https:&#x2F;&#x2F;cerebras.ai&#x2F;inference" rel="nofollow">https:&#x2F;&#x2F;cerebras.ai&#x2F;inference</a>

Understanding that there&#x27;s inherent bias by them being competitors of the other companies, but still this article seems to make some stretches. If you told me you had an 8% core defect rate reduced 100x, I&#x27;d assume you got to close to 99% enablement. The table at the end shows... Otherwise.<p>They also keep flipping between cores, SMs, dies, and maybe other block sizes. At the end of the day I&#x27;m not very impressed. They seemingly have marginally better yields despite all that effort.

I have a dumb question. Why isn&#x27;t silicon sold in cubes instead of cylinders?

Very interesting. Am I correct in saying that fault tolerance here is with respect to &#x27;static&#x27; errors that occur during manufacturing and are straightforward to detect before reaching the customer? Or can these failures potentially occur later on (and be tolerated) during the normal life of the chip?

The number of people ITT this thread who have absorbed the world-weary AI-is-a-bubble skepticism...<p>I&#x27;m just gonna say, with serene certainty,<p>the economic order we inhabit going through phase change is certain. From certain myopic perspectives we can shoehorn that into a narrative of cyclical patterns in the tech industry or financial markets etc etc.<p>This is not going to be that. No more than the transformation of American retail can be shoehorned to kind of look like it used if you don&#x27;t know anything at all about what contemporary international trade and logistics and oligopoly actually mean in terms of what is coming into your home from where and why it is or isn&#x27;t cheap.<p>Where we&#x27;ll be in 10, 20, years is literally unimaginable today; and trying to navigate that wrt traditional landmarks... oof.

How do these much smaller cores compare in computing power to the bigger ones? They seem to implicitly claim that a core is a core is a core, but surely one gets something extra out of the much bigger one?

56K mm2 vs 46K mm2. I wonder why they wouldn’t use the smart routing&#x2F;etc to use more fitting shape than square and thus use more of the wafer.

Ever heard the old joke story about an American buyer told the Japanese manufacture how many incorrectly made bolts were acceptable per lot of a thousand bolts? Maybe 2 or 3 in 1,000?<p>So the Japanese didn&#x27;t have any incorrectly made bolts in their manufacturing process so they just added two or three bad ones to every batch to please the Americans.

When I was a kid, I used to get intel keychains with a die in acrylic - good job to whoever thought of that to sell the fully defective chips.

The AMD comparison may not be accurate. The 96 core AMD CPU takes multiple such dies (eight if I remember correctly) and separate IO chiplets. The total surface area listed should be much larger.

Bear case on Cerebras: <a href="https:&#x2F;&#x2F;irrationalanalysis.substack.com&#x2F;p&#x2F;cerebras-cbrso-equity-research-report" rel="nofollow">https:&#x2F;&#x2F;irrationalanalysis.substack.com&#x2F;p&#x2F;cerebras-cbrso-equ...</a><p>Note: This author is heavily invested in Nvidia.

IIRC, it was Carl Bruggeman&#x27;s IPSA Thesis that showed us how to laser out bad cores.

Looking at the H100 on the left, why is the chip yield (72) based on a circular layout&#x2F;constraint? Why do they discard all of the other chips that fall outside the circle?

The yield problem is not surprising they found a solution. Maybe they could elaborate more on the power distribution and dissipation problem?

My biggest question is who are the buyers?

I would like a workstation with 900k cores. lmk when these things are on ebay.

Anyone has some picture how it is looks like inside these servers?

Why square chip? Make it an octagon or something.

A well written, easy to understand article.

What&#x27;s yield?

this is an important reminder that all digital electronics is really analog but with good correction circuitry.<p>and run-time cpu and memory error rates are always nonzero too, though orders of magnitude lower than chip yield rates

This is a strange blog post. Their tables say:<p>Cerebras yields 46225 * .93 = 43000 square millimeters per wafer<p>NVIDIA yields 58608 * .92 = 54000 square millimeters per wafer<p>I don&#x27;t know if their numbers are correct but it is a strange thing for a startup to brag that it is worse than a big company at something important.

&gt; Take the Nvidia H100 – a massive GPU weighing in at 814mm2. Traditionally this chip would be very difficult to yield economically. But since its cores (SMs) are fault tolerant, a manufacturing defect does not knock out the entire product. The chip physically has 144 SMs but the commercialized product only has 132 SMs active. This means the chip could suffer numerous defects across 12 SMs and still be sold as a flagship part.<p>Fault tolerance seems to be the wrong term to use here. If I wrote this, I would have written redundant.