Do Large Language Models learn world models or just surface statistics?

Huh. They&#x27;ve been able to demonstrate that the trained Othello player has a model. They did this by feeding the entire state of the trained system into a classifier. But they have no idea what the form of that model is, or how it is used to predict moves. Even for this simple, well-defined problem, it&#x27;s a total black box.<p>This level of incomprehensibility is worrisome.

As is being confirmed by this report, these large language models are absolutely building models of the real world, though not exactly at the abstraction layer level we initially guessed. The only question is how thorough of a model it is, and what is the “waste factor”, I.e., how much more inefficient are these at learning the world compared to say the human brain. It’ll be tantalizing to see how GPT-4 performs but my best guess would be that there will be an even more amazing performance but not something that can truly match human beings (even a dumb one) without significant change to the architecture.

&gt; We found that the trained Othello-GPT usually makes legal moves. The error rate is 0.01%; and for comparison, the untrained Othello-GPT has an error rate of 93.29%. This is much like the observation in our parable that the crow was announcing the next moves.<p>I read the whole article, but I think this statement largely disproves the hypothesis? We only need a single counter example to show that Othello-GPT does not have a systemic understanding of the rules, only a statistical inference of them.<p>A model that &quot;knows&quot; the rules will make no errors, a model that makes any errors does not &quot;know&quot; the rules. Simple as.<p>And I feel this way about much of the article, they state they change intermediate activations and <i>therefore</i> the observed valid results prove that the layers make up a rules engine instead of a statistical engine. And I don&#x27;t make that leap? Why would that necessarily be the case?<p>Obviously a statistical engine trained on legal moves will mostly produce legal output. A rules engine will always produce legal output.

I&#x27;m trying to figure out whether this is a fair (and, I hope, neutral) representation of what the authors have achieved here:<p>The result of training Othello-GPT is a program which, on being fed a string conforming to the syntax used to represent moves in an Othello game, outputs a usually-valid next move.<p>Through their probing of this program, the authors have gained an understanding of the program&#x27;s state after processing the input. This state can also be interpreted as representing the state of the board in the game represented by the input string.<p>This understanding allows them to make predictions about what state they would expect the program to be in if the board had reached a slightly different state (in the examples given, the changes are flips of a single disc, without regard to whether this state of play could be reached through valid moves.)<p>When the state of the program is modified to match the predicted state, it goes on to produce a move which is usually valid for the new board state.

This doesn’t seem a surprising or particularly deep result, and perhaps we’re having the wrong arguments about LLMs. I certainly don’t doubt that they’re capturing deep knowledge, my only criticism would be that they’re doing so in a lossy and extremely static way.<p>I’ve come to think of it this way. A simple Markov chain model trained on language would be the simplest example of something picking up a very shallow model of some text. Let’s call that a Layer One model, and we had fun with them for a while.<p>Machine translation goes one step further - instead of learning just the text it’s given, a model sees that above words are representations of things and ideas, and structures to render those ideas in multiple languages. That feels like Layer Two, but I’m not aware of traditional machine translation models that will take a piece of poetry, for example, and render it in a new language, perhaps using local idioms to capture the essence, if not the exact translation, of some content. So it’s only really one layer deeper.<p>Large language models are layers beyond this. They are both wider and deeper. They are showing that much more knowledge than we thought can be captured statically - patterns of patterns of patterns of patterns turn out to be enough to store and process a huge amount of what we previously thought was a privileged part of human intelligence. That’s the truly interesting result - the models still seem dumb to me, but boy, being dumb is much more impressive than we used to think.<p>We’re focused on training LLMs on larger corpora to store more stuff and more parameters and layers to find deeper patterns. But even though they’re clearly learning world models, it’s still just a static snapshot of knowledge. ChatGPT shows over a short span that a model can integrate new information quite well. What’s missing is some mechanism for _constant_ integration of new information, and some degree of internal rumination about past, present and future knowledge. Something algorithmic, temporal and dynamic, not just a static repository of knowledge. I don’t think this actually requires a great deal of magic, just a slight change in direction (although I’ve no doubt this is being worked on and is probably in all sorts of papers already). Of course, there’s still time for us to decide not to do that, but the hour is close at hand.

Btw this post is based on a preprint - might be good to wait a lil<p><a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2210.13382.pdf" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2210.13382.pdf</a>

I can barely understand the state of systems I wrote by hand.

Can any of the AIs come up with the Fosbury flop? It sounds vaguely like AlphaGo can.<p>If so, we should celebrate, panic, or do both.<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Fosbury_flop" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Fosbury_flop</a><p>Most of human &quot;progress&quot; has been done by a few, with the rest of us finishing up the details (if that). Is a goal of AI to enhance those few, or to take over for the rest of us?

This is fascinating paper looking at something really important. But I don&#x27;t really understand the Probes at all. Can someone explain it simply? Like, the classifiers are trained by showing them internal states of AIs, but where do you get the training data for the classifiers from?

I read the web page and the PDF paper quickly (<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2210.13382.pdf" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2210.13382.pdf</a>) and it seems that there are no examples of the prompts they used. It would be great to see which technique they are using or is it obvious for experts of the field?<p>It would help with reproducibility too.

I think we&#x27;re all trying to grok what LLMs like ChatGPT are really doing, and I think the answer really is that it has developed a &quot;world model&quot; for want of better words.<p>Of course an LLM is by design only trained to predict next word (based on the language statistics it has learned) so it&#x27;s tempting to say that it&#x27;s just using &quot;surface statistics&quot;, but that&#x27;s a bit too dismissive and ignores the emergent capabilities which indicates there&#x27;s rather more going on...<p>The thing is, to be a REALLY good &quot;language model&quot; you need to go well beyond grammar or short-range statistical predictions. To be a REALLY good language model you need to learn (and evidentially if based on a large enough transformer, <i>can</i> learn) about abstract contexts so that you can maintain context while generating something likely&#x2F;appropriate given all the nuances of prompt (whether that&#x27;s requesting a haiku, or python code, or a continuation of a fairy tale, etc, etc).<p>I guess one way to regard this is that it&#x27;s learnt statistical patterns on many different levels of hierarchy, and specific to many different contexts (fairy tale vs python code, etc), but of course this means that it&#x27;s representing and maintaining these (deep, hierarchical) long-range contexts while generating word-by-word output, so it seems inappropriate to just call these &quot;surface statistics&quot;, and more descriptive to refer to it as the &quot;world model&quot; it has learned.<p>One indication of the level of abstraction of this world model was shown by a recent paper which proved that the model is representing in it&#x27;s internal activations whether it&#x27;s input is true or not (correctly predicting that it will regard the negation as false), which can only reflect that this is a concept&#x2F;context is had to learn to predict well in some circumstances. For example, if generating a news story then it&#x27;s going to be best to maintain a truthy context, but for a fairly tale not so much!<p>I think how we describe, and understand, these very capable LLMs needs to go beyond their mechanics and training goals and reflect what we can deduce they&#x27;ve learned and what they are capable of. If the model is (literally) representing concepts as abstract as truth, then that seems to go far beyond what might be reasonably be called &quot;surface statistics&quot;. While I think these architectures need to be elaborated to add key capabilities needed for AGI, its perhaps also worth noting that the other impressive &quot;predictive intelligence&quot;, our brain, a wetware machine, could also be regarded as generating behavior only based on learned statistics, but at some point deep hierarchical context-dependent statistics are best called something else - a world model.

It is good that someone formulated this argument, but I have to say that the conclusion should be obvious to anyone that is not totally blinded by philosophical beliefs about intelligence and algorithms, as opposed to looking at the real evidence of the capabilities of real AI.<p>Pick any human. I claim there exists some subject that ChatGPT has a better understanding of than this human, based on a surface-level evaluation.

Did Kenneth Li publish his model? As an Othello player i would like to examine it.

I don&#x27;t think those ai are able to have reason and to understand the data they&#x27;re given. They just reshape the data without understanding it. It&#x27;s still artificial.<p>Unless there is a breakthrough in analyzing those neural blackbox, I would not hope a lot of those ai.

I think the real question is – do <i>we</i>?

Does a model learns a model, By Dr. Susse

People seem to forget about word2vec

Does a submarine swim?

The question reads like: are they lazy&#x2F;careless?

If they don&#x27;t search for Tensor path integrals, for example, can any NN or symbolic solution ever be universally sufficient?<p>A generalized solution term expression for complex quantum logarithmic relations:<p><pre><code> e**(w*(I**x)*(Pi**z)) </code></pre> What sorts of relation expression term forms do LLMs synthesize from?<p>Can [LLM XYZ] answer prompts like:<p>&quot;How far is the straight-line distance from (3<i>red, 2</i>blue, 5<i>green) to (1</i>red, 5<i>blue, 7</i>green)?&quot;<p>&gt; <i>- What are &quot;Truthiness&quot;, Confidence Intervals and Error Propagation?</i><p>&gt; <i>- What is Convergence?</i><p>&gt; <i>- What does it mean for algorithmic outputs to converge given additional parametric noise?</i><p>&gt; <i>- &quot;How certain are you that that is the correct answer?&quot;</i><p>&gt; <i>- How does [ChatGPT] handle known-to-be or presumed-to-be unsolved math and physics problems?</i><p>&gt; <i>- &quot;How do we create room-temperature superconductivity?&quot;</i><p>&quot;A solution for room temperature superconductivity using materials and energy from and on Earth&quot;<p>&gt; <i>- &quot;How will planetary orbital trajectories change in the n-body gravity problem if another dense probably interstellar mass passes through our local system?&quot;</i><p>Where will a tracer ball be after time t in a fluid simulation ((super-)fluid NDEs Non-Differential Equations) of e.g. a vortex turbine next to a stream?<p>How do General Relativity, Quantum Field Theory, Bernoulli&#x27;s, Navier Stokes, and the Standard Model explain how to read and write to points in spacetime and how do we solve gravity?