Building Meta's GenAI infrastructure

float8 got a mention! x2 more FLOPs! Also xformers has 2:4 sparsity support now so another x2? Is Llama3 gonna use like float8 + 2:4 sparsity for the MLP, so 4x H100 float16 FLOPs? Pytorch has fp8 experimental support, whilst attention is still complex to do in float8 due to precision issues, so maybe attention is in float16, and RoPE / layernorms in float16 / float32, whilst everything else is float8?

Having lived through the dot-com era, I find the AI-era slightly dispiriting because of the sheer capital cost of training models. At the start of the dot-com era, anyone could spin up an e-commerce site with relatively little infrastructure costs. Now, it seems, only the hyper-scale companies can build these AI models. Meta, Google, Microsoft, Open-AI, etc.

I know we won&#x27;t get it this from FB, but I&#x27;d be really interested to see how the relationship of compute power to engineering hours scales.<p>They mention custom building as much as they can. If FB magically has the option to 10x the compute power, would they need to re-engineer the whole stack? What about 100x? Is each of these re-writes just a re-write, or is it a whole order of magnitude more complex?<p>My technical understanding of what&#x27;s under the hood of these clusters is pretty surface level- super curious if anyone with relevant experience has thoughts?

So, I&#x27;d love to work on optimizing pipelines like this. How does one &quot;get into&quot; it? It seems a ML scientist with some C&#x2F;C++ and infra knowledge just dips down into the system when required? Or is it CUDA&#x2F;SIMD experts who move &quot;up&quot; into ML?

How much are they paying for H100&#x27;s? If they are paying $10k: 350,000 NVIDIA H100 x $10k = $3.5b

The article doesn&#x27;t mention MTIA, meta&#x27;s custom ASIC for training &amp; inference acceleration. <a href="https:&#x2F;&#x2F;ai.meta.com&#x2F;blog&#x2F;meta-training-inference-accelerator-AI-MTIA&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ai.meta.com&#x2F;blog&#x2F;meta-training-inference-accelerator...</a><p>I wonder if they will use it in RSC.

I think it’s always useful to pay attention to the history on stuff like this and it’s a rare pleasure to be able to give some pointers in the literature along with some color to those interested from first-hand experience.<p>I’d point the interested at the DLRM paper [1]: that was just after I left and I’m sad I missed it. FB got into disagg racks and SDN and stuff fairly early, and we already had half-U dual-socket SKUs with the SSD and (increasingly) even DRAM elsewhere in the rack in 2018, but we were doing <i>huge</i> NNs for recommenders and rankers even for then. I don’t know if this is considered proprietary so I’ll play it safe and just say that a click-prediction model on IG Stories in 2018 was on the order of a modest but real LLM today (at FP32!).<p>The crazy part is they were HOGWILD trained on Intel AVX-2, which is just wild to think about. When I was screwing around with CUDA kernels we were time sharing NVIDIA dev boxes, typically 2-4 people doing CUDA were splitting up a single card as late as maybe 2016. I was managing what was called “IGML Infra” when I left and was on a first-name basis with the next-gen hardware people and any NVIDIA deal was still so closely guarded I didn’t hear more than rumors about GPUs for training let alone inference.<p>350k Hopper this year, Jesus. Say what you want about Meta but don’t say they can’t pour concrete and design SKUs on a dime: best damned infrastructure folks in the game pound-for-pound to this day.<p>The talk by Thomas “tnb” Bredillet in particular I’d recommend: one of the finest hackers, mathematicians, and humans I’ve ever had the pleasure to know.<p>[1] <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;1906.00091.pdf" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;1906.00091.pdf</a><p>[2] <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2108.09373.pdf" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2108.09373.pdf</a><p>[3] <a href="https:&#x2F;&#x2F;engineering.fb.com&#x2F;2022&#x2F;10&#x2F;18&#x2F;open-source&#x2F;ocp-summit-2022-grand-teton&#x2F;" rel="nofollow">https:&#x2F;&#x2F;engineering.fb.com&#x2F;2022&#x2F;10&#x2F;18&#x2F;open-source&#x2F;ocp-summit...</a><p>[4] <a href="https:&#x2F;&#x2F;youtu.be&#x2F;lQlIwWVlPGo?si=rRbRUAXX7aM0UcVO" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;lQlIwWVlPGo?si=rRbRUAXX7aM0UcVO</a>

I wonder if Meta would ever try to compete with AWS &#x2F; MSFT &#x2F; GOOG for AI workloads

I&#x27;d be great if they could invest in an alternative to nvidia -- then, in one fell swoop, destroy the moats of everyone in the industry.

&gt; Meta’s long-term vision is to build artificial general intelligence (AGI)

350k H100 cards, around ten <i>billion</i> dollars just for the GPUs. Less if Nvidia gives a volume discount, which I imagine they do not.

Honestly Meta is consistently one of the better companies at releasing tech stack info or just open sourcing, these kinds of articles are super fun

&gt; Commitment to open AI innovation<p>I see what you did there, Meta.

Meta is still playing catch-up. Might be hard to believe but according to Reuters they&#x27;ve been trying to run AI workloads mostly on CPUs until 2022 and they had to pull the plug on the first iteration of their AI chip.<p><a href="https:&#x2F;&#x2F;www.reuters.com&#x2F;technology&#x2F;inside-metas-scramble-catch-up-ai-2023-04-25&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reuters.com&#x2F;technology&#x2F;inside-metas-scramble-cat...</a>

&gt; we have successfully used both RoCE and InfiniBand clusters for large, GenAI workloads (including our ongoing training of Llama 3 on our RoCE cluster) without any network bottlenecks.<p>Interesting dig on IB. RoCE is the right solution since it is open standards and more importantly, available without a 52+ week lead time.

This is great news for Nvidia and their stock, but are they sure the LLMs and image models will scale indefinitely? nature and biology has a preference for sigmoids. What if we find out that AGI requries different kinds of cpu capabilities

All this compute and my Instagram Reels feed still isn&#x27;t as good as my TikTok feed

&quot;Share this: Hacker News&quot; Noice

The link mentions &quot;our internal job scheduler&quot; and how they had to optimize it for this work -- does anyone know what this job scheduler is called, or how it works?

&gt; At Meta, we handle hundreds of trillions of AI model executions per day<p>Such a large number, makes sense?

&quot;Everything You Wanted to Know About GenAI at Meta, Except the One Thing You Honestly Care About&quot; (Llama 3).

it&#x27;s really interesting just how similar these systems are to the designs adopted for HPC over the past few decades. I&#x27;m salty because it took a while for the ML community to converge on this (20+K GPUs connected by a real fabric with low latency and high bandwidth).

Metas backing itself into a corner with its admirable commitment to open source. Unfortunately, at some point when they decide to monetize their billions spent and try to release a closed source model, the level of vitriol they will deal with will be an order of magnitude above what even OpenAI is experiencing. I don’t think they realize that!

Just for comparison, Swiss CSCS new Alps system will get 5k GH200 nodes (each with a H100).

Searched H100 and an Amazon link popped up. Good reviews.<p><a href="https:&#x2F;&#x2F;www.amazon.com&#x2F;Tesla-NVIDIA-Learning-Compute-Graphics&#x2F;dp&#x2F;B0C3XH4QSJ#customerReviews" rel="nofollow">https:&#x2F;&#x2F;www.amazon.com&#x2F;Tesla-NVIDIA-Learning-Compute-Graphic...</a>

You&#x27;ve got to read &quot;open&quot; roughly 3x in a paragraph.

This reads more like a flex for the investment community.

&quot;By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s.&quot; This AI game is getting into a GPU war. Heard that Meta is pushing a lot of CPU wordloads to GPU to co-locate with model inference for infra simplicity.

Subtitled &#x27;Here&#x27;s what you&#x27;ll never be able to do&#x27;.

lmfao at the Meta folks not giving any credit whatsoever to the company that actually came up with and implemented the infrastructure work.

so tired of this, not everyone need to work with AI stuff. work on facebook that is a disaster page instead

Those are some seriously great engineering numbers. Mera, with all the negative pressure it receives (rightfully so) is an engineering powerhouse.<p>But I do wonder how they foresee monetising this.

Meta seems to actually be taking all the right steps in how they&#x27;re contributing to open source AI research. Is this a &quot;commodotize your complement&quot; kind of situation?

Yann wants to be open and Mark seems happy to salt the earth.

Total cluster they say will reach 350k H100, which at $30k street price is about $10b.<p>In contrast, Microsoft is spending over $10b per quarter capex on cloud.<p>That makes Zuck look conservative after his big loss on metaverse.<p><a href="https:&#x2F;&#x2F;www.datacenterdynamics.com&#x2F;en&#x2F;news&#x2F;q3-2023-cloud-results-ai-investments-drive-up-results-and-capex&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.datacenterdynamics.com&#x2F;en&#x2F;news&#x2F;q3-2023-cloud-res...</a>