GPT4 is 8 x 220B params = 1.7T params

weird title, note that the tweet said &quot;so yes, GPT4 is *technically* 10x the size of GPT3, and all the small circle big circle memes from January were actually... in the ballpark?&quot;<p>It&#x27;s really 8 models that are 220B, which is not the same as one model that is 1.7T params. There have been 1T+ models via mixtures of experts for a while now.<p>Note also the follow up tweet: &quot;since MoE is So Hot Right Now, GLaM might be the paper to pay attention to. Google already has a 1.2T model with 64 experts, while Microsoft Bing’s modes are different mixes accordingly&quot;<p>There is also this linked tweet <a href="https:&#x2F;&#x2F;twitter.com&#x2F;LiamFedus&#x2F;status&#x2F;1536791574612303872" rel="nofollow noreferrer">https:&#x2F;&#x2F;twitter.com&#x2F;LiamFedus&#x2F;status&#x2F;1536791574612303872</a> - &quot;They are all related to Switch-Transformers and MoE. Of the 3 people on Twitter, 2 joined OpenAI. Could be related, could be unrelated&quot;<p>Which links to this tweet: &quot;Today we&#x27;re releasing all Switch Transformer models in T5X&#x2F;JAX, including the 1.6T param Switch-C and the 395B param Switch-XXL models. Pleased to have these open-sourced!&quot;<p>Anyway... remember not to just read the headlines, they can be misleading.

GPT-4 is 1.7T params in the same way that an AMD Ryzen 9 7950X is 72 GHz.

I wouldn’t trust anything geohot says. He doesn’t have access to any inside information.

I find it interesting that geohot says it is what you do “when you are out of ideas,” I can’t help but think that having multiple blended models is what makes GPT-4 seem like it has more “emergent” behavior than earlier models.

Are the models specifically trained to be experts in certain domains?<p>Or the models are all trained on the same corpus, but just queried with different parameters?<p>Is this functionally the same as beam search?<p>Do they select the best output on a token-by-token basis, or do they let each model stream to completion and then pick the best final output?

So <i>if</i> this is true - which is a big if since this looks like speculation rather than real information - could this work with even smaller models?<p>For example, what about 20 x 65B = 1.3T params? Or 100 x 13B = 1.3T params?<p>Hell, what about 5000 x 13B params? Thousands of small highly specialized models, with maybe one small &quot;categorization&quot; model as the first pass?

« We can’t really make models bigger than 220B parameters »<p>Can someone explains why?

&gt; GPT-4: 8 x 220B experts trained with different data&#x2F;task distributions and 16-iter inference.<p>There was a post on HackerNews the other day about a 13B open source model.<p>Any 220B open source models? Why or why not?<p>I wonder what the 8 categories were. I wonder what goes into identifying tokens and then trying to guess which category&#x2F;model you should look up. What if tokens go between two models, how do the models route between each other?

If it’s trivial then why does every other competitor suck at replicating it? Is it possible this is just a case of sour grapes that this intellectual is annoyed they’re not at the driving wheel of the coolest thing anymore?

That&#x27;s on the order of 25 4090 GPUs to run inference. Not a crazy number by any means. We will see consumer robots running that by the end of the decade, mark my words.

full podcast here: <a href="https:&#x2F;&#x2F;www.latent.space&#x2F;p&#x2F;geohot" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.latent.space&#x2F;p&#x2F;geohot</a>

We&#x27;re sourcing off Geohot? Yikes.

At a minimum he glossed over the multimodal capabilities of GPT-4. If they use the same set of tokens, it’s unclear how this doesn’t pollute text training data. If they use separate tokens, the model size should be bigger.

Reminds me of this: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Society_of_Mind" rel="nofollow noreferrer">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Society_of_Mind</a><p>&gt; In his book of the same name, Minsky constructs a model of human intelligence step by step, built up from the interactions of simple parts called agents, which are themselves mindless. He describes the postulated interactions as constituting a &quot;society of mind&quot;, hence the title.

I often hear the idea of digital is faster then biology. This seems mostly derived from small math computations.<p>Yet it seems the current form of large language computations is much much slower then our biology. Making it even larger will be necessary to come closer to human levels but the speed?<p>If this is the path to GI, the computational levels need to be very High and very centralized.<p>Are there ways to improve this in its current implementation other the cache &amp; more hardware?

This seems pretty consistent with what Sam Altman has said in past interviews regarding the end of continuously increasing scale and having multiple smaller specialist models: <a href="https:&#x2F;&#x2F;finance.yahoo.com&#x2F;news&#x2F;openai-sam-altman-says-giant-164924270.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;finance.yahoo.com&#x2F;news&#x2F;openai-sam-altman-says-giant-...</a>

I think this could be fake. He says it’s an MoE model, but then explains that it’s actually a blended ensemble. Anyone else have thoughts on that?

What&#x27;s 16 iter inference?

Is this still orders of magnitude smaller than a human brain?<p>How many? Based on current human neurons&#x2F;synapses knowledge?

I really wonder if it is the case that the image processing is simply more tokens appended to the sequence. Would make the most sense from an architecture perspective, training must be a whole other ballgame of alchemy though

I like how geohot takes these concepts that seem impossibly complex to an outsider (mixture of models, multi modality) and discusses them so casually that they seem accessible to anyone. The flippancy is refreshing.

Interesting. And makes sense. Eg I could see one of the eight being very close focussed and trained on GitHub like data. Could help it stay on tasks too

Sad that there is a grammar error lining the top of the video.