The article talks about a few things that they call inventions, like making interconnections across what would normally be scribe lines. But I personally worked on wafer scale integration about 25 years ago and we were already doing that. We called it inter-reticle stitching. The technology was ancient back then - 0.5 micron feature size on 4 inch wafers - but the wafer scale techniques are applicable to modern technologies. In particular, developing a yield model that informs your on-chip redundancy choices and designing built-in self test and selection circuitry so that you can yield large chips. The chip we developed was so large that only two would fit on a wafer. We got 50% yield on a line that was far from mature at the time. The company lacked the vision to do anything with what they had developed. To them it was just a chip for which there were few customers. The suits didn't know how to make bank with this methodology that could yield nearly arbitrarily complex chips in nearly any target process.<p>Edit: There were a number of papers and conference proceedings published back then but not much shows up when searching Google. Here's one discussing the issues and results of field stitching <a href="https://fdocuments.in/document/ieee-comput-soc-press-1992-international-conference-on-wafer-scale-integration-589dfe172703a.html" rel="nofollow">https://fdocuments.in/document/ieee-comput-soc-press-1992-in...</a><p>From 1992, so yeah, field stitching is not a recent invention.
I don’t know if this mega-chip will be successful, but I like the idea. Before I retired I managed a deep learning team that had a very cool internal product for running distributed TensorFlow. Now in retirement I get by with a single 1070 GPU for experiments - not bad but having something much cheaper, much more memory, and much faster would help so much.<p>I tend to be optimistic, so take my prediction with a grain of salt: I bet within 7 or 8 years there will be an inexpensive device that will blow away what we have now. There are so many applications for much larger end to end models that will but pressure on the market for something much better than what we have now. BTW, the ability to efficiently run models on my new iPhone 11 Pro is impressive and I have to wonder if the market for super fast hardware for training models might match the smartphone market. For this to happen, we need a deep learning rules the world shift. BTW, off topic, but I don’t think deep learning gets us to AGI.
I am far more excited by the underlying Wafer Scale Integration moonshot than I am by any AI benchmarks here. I know it's trendy to think there can only be one w/r to the AI Iron Throne but nope, not the case, everyone is writing bespoke code in production where the money is made. Well, almost everyone, Amazon seems to be the odd duck but they're a bunch of cheapskate thought leaders anyway (except for their offers to junior engineers in their desperate hail mary attempt to catch up with FAIR and DeepMind, but... I... digress...).<p>Which is to say that graphs written to run specifically on Cerebras's giant chip will smash deep learning's speed barrier for graphs written to run best on Cerebras's giant chip. And that's great, but it won't be every graph, there is no free lunch. Hear me now, believe me later(tm).<p>But if we can cut the cost of interconnect by putting a figurative datacenter's worth of processors on a chip, that's genuinely interesting, and it has applications far beyond the multiplies and adds of AI. But be very wary of anyone wielding the term "sparse" for it is a massively overloaded definition and every single one of those definitions is a beautiful and unique snowflake w/r to efficient execution on bespoke HW.
Reminds me of that other great prediction of a GPU killer from IEEE Spectrum back in 2009:<p><a href="https://spectrum.ieee.org/computing/software/winner-multicore-made-simple" rel="nofollow">https://spectrum.ieee.org/computing/software/winner-multicor...</a>
A chip that size, imagine the yield. Equally, cooling - has to be water based as a heatsink that size would be on par to a small anvil and the weight factor would be some serious issues. Though unsure as no pictures of it in-play alas and all they say is - "20 kilowatts being consumed by each blew out into the Silicon Valley streets through a hole cut into the wall", which does somewhat beg for a picture as just raises more questions.<p>Why would they make a chip this big with AMD showing a chiplet design approach is cheaper and more scalable on so many levels. Let alone, yields.<p>Equally, arms approach to utilising the back of the chip as a power delivery :- <a href="https://spectrum.ieee.org/nanoclast/semiconductors/design/arm-shows-backside-power-delivery-as-path-to-further-moores-law" rel="nofollow">https://spectrum.ieee.org/nanoclast/semiconductors/design/ar...</a><p>Then a wafer scale chip like this, using that approach, would save so much power. But again, yeilds will be a factor and can imagine this is not the cutting edge process node as you find as nodes mature, the yields improve. So an older node size would have a better yield and be more suitable for such wafer scale chips. But again, no mention of what is used. I have read in the past that it would use Intel's 10nm, but this article mentions TSMC. Another article I read that they used a 16nm node ( <a href="https://fuse.wikichip.org/news/3010/a-look-at-cerebras-wafer-scale-engine-half-square-foot-silicon-chip/" rel="nofollow">https://fuse.wikichip.org/news/3010/a-look-at-cerebras-wafer...</a> ), which as mentioned above about node maturity, understandable.
From the perspective of an outsider, I can't see how a company like this could survive. They claim on the one hand to have done something really amazing and are at the stage where they are looking for customers. Normally, you'd expect them to be touting performance figures to secure such investment. Instead, they've decided to keep the performance secret. And they've managed to find some "expert" who says this is normal.<p>Does anyone here have expertise in this area? Is this the model for a successful company in this area?
The members of the GraphBLAS forum have discussed this chip a couple of times. There's a lot of research on making deep neural networks more sparse, not just by pruning a dense matrix, but by starting with a sparse matrix structure de novo. Lincoln Laboratory's Dr. Jeremy Kepner has a good paper on Radix-Net mixed radix topologies that achieve good learning ability but with far fewer neurons and memory requirements. Cited in the paper was a network constructed with these techniques that simulated the size and sparsity of the human brain:<p><a href="https://arxiv.org/pdf/1905.00416.pdf" rel="nofollow">https://arxiv.org/pdf/1905.00416.pdf</a><p>It would be cool to see the GraphBLAS API ported to this chip, which from what I can tell comes with sparse matrix processing units. As networks become bigger, deeper, but sparser, a chip like this will have some demonstrable advantages over dense numeric processors like GPUs.
I've wrote about the challenges that Cerebras went through and what is next: <a href="https://towardsdatascience.com/why-cerebras-announcement-is-a-big-deal-6c8633ffc49c" rel="nofollow">https://towardsdatascience.com/why-cerebras-announcement-is-...</a>
This fits perfectly into the narrative of yesterday's discussion on HN [1].<p>Deep Neural Nets are somewhat of a brute force approach to machine learning. Training efficiency is horrible as compared with other ML approaches, but hey, as long as we can trade +5% of classification performance for +500% of NN complexity and throw more money at the problem, who cares?<p>I see a dystopian future where much better and much more efficient approaches to ML exist, but nobody's paying attention because we have Deep Neural Nets in hardware and decades of infrastructure supporting it.<p>[1] <a href="https://news.ycombinator.com/item?id=21929709" rel="nofollow">https://news.ycombinator.com/item?id=21929709</a>
They did build some valuable tech, no question there, but be sure to account for the typical startup hyperbole. By the time you can get your hands on this (if that ever happens), the hyperbole will converge a bit closer to reality, the tradeoffs will become apparent, etc, and you'll discover that it is not, in fact, going to "smash" barriers of any kind in any practical sense.<p>From TFA: "Cerebras hasn’t released MLPerf results or any other independently verifiable apples-to-apples comparisons."<p>That's all you really need to know.
The Cerebras chip really stands out in terms of the chip industry's relationship to Moore's law. Look at the graphs in this article for reference:<p><a href="https://medium.com/predict/cerebras-trounces-moores-law-with-first-working-wafer-scale-chip-70b712d676d0" rel="nofollow">https://medium.com/predict/cerebras-trounces-moores-law-with...</a>