Procedural knowledge in pretraining drives reasoning in large language models

It seems obvious to me that LLMs wouldn&#x27;t be able to find examples of every single problem posed to them in training data. There wouldn&#x27;t be enough examples for the factual look up needed in an information retrieval style search. I can believe that they&#x27;re doing some form of extrapolation to create novel solutions to posed problems.<p>It&#x27;s interesting that this paper doesn&#x27;t contradict the conclusions of the Apple LLM paper[0], where prompts were corrupted to force the LLM into making errors. I can also believe that LLMs can only make small deviations from existing example solutions in creation of these novel solutions.<p>I hate that we&#x27;re using the term &quot;reasoning&quot; for this solution generation process. It&#x27;s a term coined by LLM companies to evoke an almost emotional response on how we talk about this technology. However, it does appear that we are capable of instructing machines to follow a series of steps using natural language, with some degree of ambiguity. That in of itself is a huge stride forward.<p>[0] <a href="https:&#x2F;&#x2F;machinelearning.apple.com&#x2F;research&#x2F;gsm-symbolic" rel="nofollow">https:&#x2F;&#x2F;machinelearning.apple.com&#x2F;research&#x2F;gsm-symbolic</a>

You mean you need humans to step-by-step solve a problem so a neural net can mimic it? It sounds kinda obvious now that I write it out.

This would explain the unexpected benefits of training on code.

&gt;On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies<p>surprised this gets voted up given the surprising amount of users on HN who think LLMs can&#x27;t reason at all and that the only way to characterize an LLM is through the lens of a next token predictor. Last time I was talking about LLM intelligence someone rudely told me to read up on how LLMs work and that we already know exactly how they work and they&#x27;re just token predictors.

This is highly relevant to the recent discussion at <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=42285128">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=42285128</a>.<p>Google claims that their use of pretraining is a key requirement for being able to deliver a (slightly) better chip design. And they claim that a responding paper that did not attempt to do pretraining, should have been expected to be well below the state of the art in chip design.<p>Given how important reasoning is for chip design, and given how important pretraining is for driving reasoning in large language models, it is obvious that Google&#x27;s reasoning is very reasonable. If Google barely beats the state of the art while using pretraining, an attempt that doesn&#x27;t pretrain should be expected to be well below the current state of the art. And therefore that second attempt&#x27;s poor performance says nothing about whether Google&#x27;s results are plausible.

&gt; In the extreme case, a language model answering reasoning questions may rely heavily on retrieval from parametric knowledge influenced by a limited set of documents within its pretraining data. In this scenario, specific documents containing the information to be retrieved (i.e. the reasoning traces) contribute significantly to the model’s output, while many other documents play a minimal role.<p>&gt; Conversely, at the other end of the spectrum, the model may draw from a broad range of documents that are more abstractly related to the question, with each document influencing many different questions similarly, but contributing a relatively small amount to the final output. We propose generalisable reasoning should look like the latter strategy.<p>Isn&#x27;t it much more impressive if a model can generalize from a single example?

that resonates - less facts and more reasoning training data. The most low hanging in terms of non synthetic data probably being mathematical proofs. With prolog and the like many alternate reasoning paths could be generated. It&#x27;s hard to say if these many-path would help in llm training without access to the gigantic machines (it&#x27;s so unfair) to try it on.

Is this conclusion similar to my layman&#x27;s understanding of AlphaGo vs AlphaZero? That human procedural knowledge helps ML training to a point, and from there on becomes a limitation?

Does this mean LLMs might do better if trained on large amounts of student notes, exams, book reviews and such? That would be incredibly interesting.

Okay dumb question, why are the images they generate nightmarish nonsense. Why can’t they procedurally construct a diagram

drives retrieval of patterns of procedure?<p>I mean - like for arithmetic?

Going meta a bit: comments so far on this post show diametrically opposing understandings of the paper, which demonstrates just how varied the interpretation of complex text can be.<p>We hold AI to a pretty high standard of correctness, as we should, but humans are not that reliable on matters of fact, let alone on rigor of reasoning.

thanks