Recent results show that LLMs struggle with compositional tasks

The main thesis here seems to be that LLMs behave like almost all other machine learning models, in that they are doing pattern matching on their input data, and short circuiting to a statistically likely result. Chain of thought reasoning is still bound by this basic property of reflexive pattern matching, except the LLM is forced to go through a process of iteratively refining the domain it does matching on.<p>Chain of thought is interesting, because you can combine it with reinforcement learning to get models to solve (seemingly) arbitrarily hard problems. This comes with the caveat that you need some reward model for all RL. This means you need a clear definition of success, and some way of rewarding being closer to success, to actually solve those problems.<p>Framing transformer based models as pattern matchers makes all the sense in the world. Pattern matching is obviously vital to human problem solving skills too. Interesting to think about what structures human intelligence has that these models don&#x27;t. For one, humans can integrate absolutely gargantuan amounts of information extremely efficiently.

LLMs keep showing more and more they are the wonder of AI that we awaited for decades: talking machines that every two months do progresses that two months before were anticipated impossible because of &lt;put here some limit that was actually in the prejudice of the skeptical AI community&gt; (just stochastic parrots, no reasoning possible without symbolic representations, there are no longer tokens, ...)<p>At the same time, part of the scientific community continues to diminish what was accomplished and the steps that are being made. A few months ago LeCun arrived to tell new researchers to move away from LLMs since they are a dead end: imagine the disservice he made to the surely non-zero folks that followed the advice, putting themselves out of the AI research that matters. (Incidentally, this skepticism of the Meta AI head must have something to do with the fact that Meta, despite the huge efforts allocated, produced the worst LLM among Anthropic, OpenAI, DeepSeek -- I bet Zuckerberg is asking questions lately).<p>It&#x27;s very hard to explain this behavior if not by psychological denial.<p>[EDIT: you can&#x27;t see the score of this comment, but I can: it&#x27;s incredible how it goes from 3, to -2, to 1, and so forth. The community is split in two, and it is pretty sad since this is not a matter of taste or political inclination: there must be a single truth]

What a poorly informed article. It&#x27;s very shallow and out of touch with LLM research. As it stands 6-12 months old models are system 1 thinkers, everybody knows this and knew this even at the time. You need system 2 thinking (test time compute) for more complex logical, algorithmic and reasoning tasks. We knew this when Daniel kahneman wrote thinking fast, thinking slow (over a decade ago) and we still know it today. So LLMs can think but they have to be programmed to think (a la system 2, reasoning, thinking models). There&#x27;s nothing inherently wrong or limited with LLMs themselves as far as we can tell.

Idk, `o3-mini-high` was able to pop this Prolog code out in about 20 seconds:<p><pre><code> solve(WaterDrinker, ZebraOwner) :- % H01: Five houses with positions 1..5. Houses = [ house(1, _, norwegian, _, _, _), % H10: Norwegian lives in the first house. house(2, blue, _, _, _, _), % H15: Since the Norwegian lives next to the blue house, house(3, _, _, milk, _, _), % and house1 is Norwegian, house2 must be blue. house(4, _, _, _, _, _), house(5, _, _, _, _, _) ], % H02: The Englishman lives in the red house. member(house(_, red, englishman, _, _, _), Houses), % H03: The Spaniard owns the dog. member(house(_, _, spaniard, _, dog, _), Houses), % H04: Coffee is drunk in the green house. member(house(_, green, _, coffee, _, _), Houses), % H05: The Ukrainian drinks tea. member(house(_, _, ukrainian, tea, _, _), Houses), % H06: The green house is immediately to the right of the ivory house. right_of(house(_, green, _, _, _, _), house(_, ivory, _, _, _, _), Houses), % H07: The Old Gold smoker owns snails. member(house(_, _, _, _, snails, old_gold), Houses), % H08: Kools are smoked in the yellow house. member(house(_, yellow, _, _, _, kools), Houses), % H11: The man who smokes Chesterfields lives in the house next to the man with the fox. next_to(house(_, _, _, _, _, chesterfields), house(_, _, _, _, fox, _), Houses), % H12: Kools are smoked in a house next to the house where the horse is kept. next_to(house(_, _, _, _, horse, _), house(_, _, _, _, _, kools), Houses), % H13: The Lucky Strike smoker drinks orange juice. member(house(_, _, _, orange_juice, _, lucky_strike), Houses), % H14: The Japanese smokes Parliaments. member(house(_, _, japanese, _, _, parliaments), Houses), % (H09 is built in: Milk is drunk in the middle house, i.e. house3.) % Finally, find out: % Q1: Who drinks water? member(house(_, _, WaterDrinker, water, _, _), Houses), % Q2: Who owns the zebra? member(house(_, _, ZebraOwner, _, zebra, _), Houses). right_of(Right, Left, Houses) :- nextto(Left, Right, Houses). next_to(X, Y, Houses) :- nextto(X, Y, Houses); nextto(Y, X, Houses). </code></pre> Seems ok to me.<p><pre><code> ?- solve(WaterDrinker, ZebraOwner). WaterDrinker = norwegian, ZebraOwner = japanese .</code></pre>

There&#x27;s so much talk about the advancements in AI&#x2F;LLMs, yet for me ChatGPT as of this date is basically just a faster search engine without cookie banners, clickbait and ads. It hallucinates a lot and it can keep very limited context. Why is there so much promise about future progress but so little actual progress?

One must now ask whether research results are analyzing pure LLMs (eg. gpt-series) or LLM synthesis engines (eg. o-series, r-series). In this case, the headline is summarizing a paper originally published in 2023 and does not necessarily have bearing on new synthesis engines. In fact, evidence strongly suggests the opposite given o3&#x27;s significant performance on ARC-AGI-1 which requires on-the-fly composition capability.

I&#x27;ve managed to get llms fail on simple questions, that require thinking graphically - 2D or 3D.<p>An example would be: you have a NxM grid. How many shapes of XYZ shape can you fit on it?<p>However, thinking of the transformer video games, AI can be trained to have a good representation of 2D&#x2F;3D worlds. I wonder how it can be combined so that this graphical representation is used to compute text output.

When one of these limitations gets spelled out in an article, it feels like six months later, somebody has a demo of a chatbot without that particular limitation.<p>These limitations don’t seem in any way “fundamental” to me. I’m sure there are a ton of people gluing LLMs to SAT solvers as we speak.

By the time these academic studies get published they are usually already several months out of date. o3-mini was released yesterday and if one wants to know about the limitations of current technology they are much better to check twitter than some research paper

<a href="https:&#x2F;&#x2F;chatgpt.com&#x2F;share&#x2F;679f0353-12bc-8007-91cf-dd63d52044a0" rel="nofollow">https:&#x2F;&#x2F;chatgpt.com&#x2F;share&#x2F;679f0353-12bc-8007-91cf-dd63d52044...</a><p>O1 Pro gets the answer with proper reasoning<p>Can’t tell if this is dude to data contamination or it really figured it out?<p>How can we form the question in another way to avoid data contamination?

&quot;Recent results&quot; = two year old study looking at GPT-3, 3.5, and first gen 4.

Perhaps applying:<p><a href="https:&#x2F;&#x2F;www.doc.ic.ac.uk&#x2F;~re14&#x2F;Evans-R-2020-PhD-Thesis.pdf" rel="nofollow">https:&#x2F;&#x2F;www.doc.ic.ac.uk&#x2F;~re14&#x2F;Evans-R-2020-PhD-Thesis.pdf</a><p>&quot;Kant’s Cognitive Architecture&quot;<p>will help?<p>&gt; ...we provide a precise formalization of what it means<p>&gt; to “make sense” of a sensory sequence. According to our definition, making sense means constructing<p>&gt; a symbolic causal theory that explains the sensory sequence and satisfies a set of unity conditions<p>&gt; that were inspired by Kant’s discussion in the first half of the Critique of Pure Reason. According to<p>&gt; our interpretation, making sense of sensory input is a type of program synthesis, but it is unsupervised<p>&gt; program synthesis.

&gt; Chatbot Software Begins to Face Fundamental Limitations<p>&gt; Recent results show that large language models struggle with compositional tasks, suggesting a hard limit to their abilities.<p>Your first question with anything like this should always be <i>WHICH MODELS</i>:<p>&gt; For our experiments, we evaluate the performance of 6 LLMs: GPT4 (gpt-4) [58], ChatGPT (GPT3.5-turbo) [57], GPT3 (text-davinci-003) [11], FlanT5 [17] and LLaMa [75].<p>This is <i>ancient</i>. This research was done <i>centuries</i> ago. This is research about the possibility of isotopes, written about radium in 1903, published in 1946. It is a criminal level of journalistic malpractice to leave uninformed readers with the impression that this is where AI stands <i>yesterday</i>.

I&#x27;d like to propose a modified Zebra puzzle. Still has unique solutions. Not on the internet, so not part of the training set. Similar problem description complexity. Just remove the last constraint (Norwegian lives next to the blue house) and asked negated questions: Who cannot be the water drinker? Who cannot own the zebra? Text of problem here: sandiway.arizona.edu&#x2F;mzebra.pdf (I tested it on free ChatGPT and DeepSeek-R1 <a href="https:&#x2F;&#x2F;youtu.be&#x2F;3gotauWUcew" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;3gotauWUcew</a>)

This is ultimately a basic adaptation of the pigeonhole principle and is not surprising. A finite system of matrix multiplications cannot be turing complete. You cannot expect one trip through a series of matrix multiplications and bias additions and a final sampling at the end which commits it to a certain answer to ever produce a correct answer. It&#x27;s a mathematical impossibility. No talk of quantum woo, emergent phenomenen, or whatever other pseudo-science has arisen to explain AI intelligence can get around this simple truth of mathematics.<p>However, chain of thought reasoning where token streams can continue ad infinitum could potentially solve large swaths of problems whose ordinary solutions require turing machines. It could also solve problems that cannot generally be solved by turing machines, but where you only need solutions for a few classes of problems.<p>Either way, even with chain of thought, you would expect that... in some instances, the model output diverges and does not complete. And unsurprisingly... this is exactly what you see with the DeepSeek models (and other CoT models) when you pose it difficult questions. It will never emit the &lt;&#x2F;think&gt; tag.

I&#x27;m a software developer with just a basic understanding of LLMs.<p>But its not surprising to me that LLMs wouldnt be good at composition and planning. To me, LLMs seem like half a brain, the right half, that does pattern matching and such.<p>To me, the current crop of AI seems to be missing the other half of the brain, the part that&#x27;s good at planning and composition and such.<p>LLMs can tell me all I want to know about logic and deduction, so its funny that LLMs cant understand their own words.

&gt; multilayer transformers indeed cannot solve certain complicated compositional tasks<p>&gt; chain-of-thought prompting essentially turns a large problem into a sequence of smaller problems, making it possible for transformers to tackle more complex compositional tasks<p>--- [out of order]<p>&gt; the model could be trained on 20-digit numbers and still reliably (with 98% accuracy) add 100-digit numbers, whereas a model trained without the extra positional embedding was only about 3% accurate

Maybe they’re not training them on the right thing, maybe math can be a different “mode” just lime sound or images

Maybe it&#x27;s just me but when i code&#x2F;program i typically think of how different variations of implementations would affect roi of the associated company i would be working for.<p>That&#x27;s something that AI in the current state probably wouldn&#x27;t be able to take into account.

Isn&#x27;t the math calculation thing solvable through the integration with a simple Python script for basic math and more complex stuff something like an open source version of Wolfram?

&quot;Fundamental&quot; results cannot come from empirical findings, they have to be proven rigorously. Until then, they are not fundamental

Here&#x27;s &quot;Einstein’s puzzle&quot; from this paper: <a href="https:&#x2F;&#x2F;www.researchgate.net&#x2F;publication&#x2F;341189675_Is_Einstein&#x27;s_Puzzle_Over-Specified" rel="nofollow">https:&#x2F;&#x2F;www.researchgate.net&#x2F;publication&#x2F;341189675_Is_Einste...</a><p><pre><code> H01 There are five houses. H02 The Englishman lives in the red house. H03 The Spaniard owns the dog. H04 Coffee is drunk in the green house. H05 The Ukrainian drinks tea. H06 The green house is immediately to the right of the ivory house. H07 The Old Gold smoker owns snails. H08 Kools are smoked in the yellow house. H09 Milk is drunk in the middle house. H10 The Norwegian lives in the first house. H11 The man who smokes Chesterfields lives in the house next to the man with the fox. H12 Kools are smoked in a house next to the house where the horse is kept. H13 The Lucky Strike smoker drinks orange juice. H14 The Japanese smokes Parliaments. H15 The Norwegian lives next to the blue house. Now, Q1 Who drinks water? Q2 Who owns the zebra?</code></pre>

&quot;showing that multilayer transformers indeed cannot solve certain complicated compositional tasks. Basically, some compositional problems will always be beyond the ability of transformer-based LLMs.&quot;<p>Pretty sure this is just false and the paper doesn&#x27;t show this. I could be misunderstanding, but it looks like the result is only about a single token&#x2F;forward pass, not a reasoning model with many thousands of tokens like o1&#x2F;o3

Why do articles testing ancient LLMs keep getting posted?

Quite deep and informative. One thing to point out here is how they measure accuracy when testing their LLMs. Like most neural networks, the n-dimensional space of LLMs is largely sparse. In simple terms, this means that it’s very easy to “exercise” the model in areas where it’s weak. That doesn’t mean that the model doesn’t have the potential of doing a (much) better job if you “exercise” its space properly—which is exactly what techniques like CoT basically do. LLMs are inherently limited, and they don’t have “true” reasoning capabilities—beyond marketing hype, this much should be obvious to most serious ML researchers today. The only question is how well we can get them to “mimic” reasoning for practical applications, and this is where “prompt engineering”, strictly speaking, is a true form of engineering, which has to take into account the mathematical foundations of the models and figure out how to extract out of them the best performance they can deliver.

It feels like every few weeks someone puts out a new article breathlessly talking about how bad the fancy autocorrect bot is at math and logic like it&#x27;s brand new information. How have we as a society not gotten this through our skulls yet?

Do you really need a study to work out that a Markov chain can’t solve problem questions? It feels pretty intuitive.<p>LLMs are not intelligent. Human intelligence is stored in text, so mimicking text mimicks that intelligence. The sheer quantity of text means they’ve probably seen something similar to the prompt.<p>If LLMs were an intelligent system they wouldn’t need to steal 1000TB of text and media content. Have you ever seen a person require a million books? It shows that the learning is brute force rather than intelligent.<p>It’s close, but it’s just not there. The methodology is wrong.