30% drop in O1-preview accuracy when Putnam problems are slightly variated

I remember when this stuff was all coming out and people were finally excited about ChatGPT getting the problem with &quot;which is heavier, a 10 pound bag of feathers or a 10 pound bag of bricks?&quot; problem correct. But of course it got it correct. It was in the training set. Vary the problem slightly by just changing the nouns, or changing the numbers so that one in fact was heavier than the other, and performance went all over the map.<p>I just went to chatgpt.com and put into the chat box &quot;Which is heavier, a 9.99-pound back of steel ingots or a 10.01 bag of fluffy cotton?&quot;, and the very first answer I got (that is, I didn&#x27;t go fishing here) was<p><pre><code> The 9.99-pound bag of steel ingots is heavier than the 10.01-pound bag of fluffy cotton by a small margin. Although the cotton may appear larger due to its fluffy nature, the steel ingots are denser and the weight of the steel bag is 9.99 pounds compared to the 10.01 pounds of cotton. So, the fluffy cotton weighs just a tiny bit more than the steel ingots. </code></pre> Which, despite getting it both right and wrong, must still be graded as a &quot;fail&quot;.<p>If you want to analyze these thing for their true capability, you <i>need</i> to make sure you&#x27;re out of the training set... and most of the things that leap to your mind in 5 seconds are leaping to your mind precisely because they are either something you&#x27;ve seen quite often or something that you can easily think of and therefore many other people have easily thought of them as well. Get off the beaten path a bit and the math gets much less impressive.

One experiment I would love to see, although not really feasible in practice, is to train a model on <i>all</i> digitized data from before the year 1905 (journals, letters, books, broadcasts, lectures, the works), and then ask it for a formula for mass-energy equivalence. A certain answer would definitely settle the debate on whether pattern recognition is a form of intelligence ;)

Performance of these LLMs on real life tasks feels very much like students last-minute cramming for Asian style exams.<p>The ability to perfectly regurgitate, while no concept of meaning.

I have a feeling the fact you&#x27;re only slightly varying the input means the model is falling back into the question it was expecting and getting things wrong as a result. If you just varied it a little <i>more</i> and added some general-purpose prompt-fu like:<p>&quot;First break the problem down into known facts, then pull relevant world knowledge, then bring it all together to assess the problem from multiple angles and make a conclusion. Do not immediately just use the first obvious conclusion.&quot;<p>You&#x27;re gonna get a lot better responses. I suspect this is more of a &quot;look! LLMs make bad kneejerk responses when we try to trick them from what they were expecting!&quot; rather than &quot;Look! They aren&#x27;t even smart reasoners, they can&#x27;t even figure out these problems without memorizing!&quot;<p>They do memorize. But that cuts both ways - making problems very close to the memorized one mess with their perception, the same way humans will instinctually respond to something that looks like a face before stepping back and assessing.

This is very interesting, but a couple of things to note; 1. o1 still achieves &gt; 40% on the varied Putnam problems, which is still a feat most math students would not achieve. 2. o3 solved 25% of the Epoch AI dataset. - There was an interesting post which calls into question how difficult some of those problems actually are, but it still seems very impressive.<p>I think a fair conclusion here is reasoning models are still really good at solving very difficult math and competitive programming problems, but just better at ones they have seen before.

Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to a chatbot :&#x2F;

They are highly effective pattern matchers. You change the pattern, it won&#x27;t work. I don&#x27;t remember who, but most likely @tszzl (roon), commented on x that they still trained the traditional way, and there is no test time compute (TTC) or Montecarlo Tree search (like Alpha Go) in o1 or o3. If that is true, then it&#x27;s still predicting the next word based on it&#x27;s training data. Likely to follow the most probable path - which comes directly from the training itself - even for the slight variations. Encouragingly, if TTC hasnt been explored, there is a long runway for performance improvements.<p>The other reason this seems hard to guess is because we don&#x27;t know how much of what we are asking is in the training data. It would perform on some tasks, while fail at others even though those are similar.

This workshop contribution is OK, and the benchmark is somewhat valuable even without the rephrasing part of the problems, but the rephrasing (of only a small number of problems) sometimes genuinely makes the problem more confusing to humans as well by either poor phrasing (fig 3), or unneeded breaking of convention (fig 4; points in 2D are often P, with coordinates x,y). It would have been nice to see effects on the rephrasing of the latest&#x2F;post-training date problems as a function of the increased noising, to delineate part of this confusion. I wonder how much better o3 is on the same benchmark.<p>Also, the correct title of this contribution is: Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning

The paper includes several examples of their modified questions. There has been a substantial jump from o1-preview to o1, so I gave several samples to o1 and o1-pro (<i>not</i> o1-preview), and current o1s gave the correct answer to those modified problems. SOTA changes fast.

So many negative comments as if o3 didn’t get <i>25% on frontiermath</i> - which is absolutely nuts.<p>Sure, LLMs will perform better if the answer to a problem is directly in their training set. But that doesn’t mean they perform <i>bad</i> when the answer isn’t in their training set.

I don&#x27;t get why this matters at all? I looked at the o1-preview paper, Putnam is not mentioned. Meaning that: a) OpenAI never claimed this model achieves X% on this dataset. b) likely, OpenAI did not take measures to exclude this dataset from training. Meaning the only conclusion we can draw from this result is: when prompted with questions that were verbarim in the dataset, performance increases dramatically. We already know this, and it doesn&#x27;t say anything about the performance of the model on unseen problems.

An interesting example of this is:<p>There are 6 “a”s in the sentence: “How many ‘a’ in this sentence?”<p><a href="https:&#x2F;&#x2F;chatgpt.com&#x2F;share&#x2F;677582a9-45fc-8003-8114-edd2e6efa289" rel="nofollow">https:&#x2F;&#x2F;chatgpt.com&#x2F;share&#x2F;677582a9-45fc-8003-8114-edd2e6efa2...</a><p>Whereas the typical “strawberry” variant is now correct.<p>There are 3 “r”s in the word “strawberry.”<p>Clearly the lesson wasn’t learned, the model was just trained on people highlighting this failure case.

I wouldn’t be surprised if similar will be found concerning the ARC challenge and it is why I still maintain my own private LLM challenges to gauge current capabilities. Course, I have little illusion that these are fully private, but it is better than fully public tests.<p>Even the most straight forward, logical, easily reasoned ones stump all LLMs I have access to, which is why I am so skeptical concerning emergence, reasoning and all this hype around “AGI”…

The metaphor that might describe this paper is &quot;iteration&quot;. I&#x27;d hazard to predict that we’ll likely see more iterations of the following loop in 2025:<p>-&gt; A new benchmark emerges with a novel evaluation method.<p>-&gt; A new model saturates the benchmark by acquiring the novel “behavior.”<p>-&gt; A new benchmark introduces yet another layer of novelty.<p>-&gt; Models initially fail until a lab discovers how to acquire the new behavior.<p>Case in point: OpenAI tackled this last step by introducing a paradigm called deliberative alignment to tackle some of the ARC benchmarks. [1]<p>Alongside all this technical iteration, there’s a parallel cycle of product iteration, aiming to generate $ by selling intelligent software. The trillion $ questions are around finding the right iterations on both technical and product dimensions.<p>[1] <a href="https:&#x2F;&#x2F;openai.com&#x2F;index&#x2F;deliberative-alignment&#x2F;" rel="nofollow">https:&#x2F;&#x2F;openai.com&#x2F;index&#x2F;deliberative-alignment&#x2F;</a>

ok but preview sucks, run it on o1 pro.<p>99% of studies claiming some out of distribution failure of an LLM uses a model already made irrelevant by SOTA. These kinds of studies, with long throughputs and review periods, are not the best format to make salient points given the speed at which the SOTA horizon progresses

One I just did:<p>Q: I was heading to Aberdeen from London. On my way I passed seven wives, each wife had seven sacks, and in each sack there were seven cats and each cat had seven fish. How many were going to London?<p>A: This riddle is a play on words, and the answer is hidden in the phrasing! You mentioned you were heading to Aberdeen from London, but you didn’t say anything about the seven wives, their sacks, cats, or fish actually being headed to London.<p>The only one going to London is you.<p>So the answer is: 1 person (you) are going to London.

The researcher&#x27;s answer to their variant of &quot;Year: 2016 ID: A1&quot; in the appendix is wrong.<p>The solution (sum of 1,2,5,6,9,10,13,14, ...) has an alternating pattern, so has to be two piecewise interleaved polynomials, which cannot be expressed as a single polyomial.<p>Their answer works for k=1,2, but not k=3.<p><a href="https:&#x2F;&#x2F;openreview.net&#x2F;pdf?id=YXnwlZe0yf" rel="nofollow">https:&#x2F;&#x2F;openreview.net&#x2F;pdf?id=YXnwlZe0yf</a><p>This does not give me confidence in the results of their paper.

Link title says &quot;slightly&quot;, but the PDF says two different kinds of variations: variable names (slight) and problem constants (significant), and the 30% drop is on the combination of a 26 variable and also 26 variable + constant questions.<p>It&#x27;s good to have a better test (though I bet this one will also be quickly saturated like all the others), but the title here doesn&#x27;t seem justified by the page title there or the content.

This result is the same as a recent test of the same method+hypothesis from a group at Apple, no? I don’t have that reference handy but I don’t think I’m making it up.

Isn&#x27;t this simply because the dataset used (Putnam-AXIOM Original) is in the training data used to train the various models?<p>Given that these are simple variations (variable names and constants value change in math problems). Why would the companies creating these models (OpenAI etc.) create these variations themselves in order to insure that the model is learning how to solve the problem rather than memorize a solution? Seems like a very obvious thing to do ...

I still find it hard to believe that LLM methods will lead to &quot;true&quot; AI. No amount of processing power or data will be sufficient without something new.

I don&#x27;t think this proves that the LLM is just &quot;pattern matcher&quot;. Human makes similar mistakes too, especially when under time pressure (similar to non-reasoning model that needs to &quot;use system one&quot; to generate answer on one go). This is further evident that if you specifically ask the models to pay attention to traps, or just ask follow up question &quot;are you sure?&quot;, then they usually can get it right.

I hope someone reruns this on o1 and eventually o3.<p>If o1-preview was the start like gpt1, then we should expect generalization to increase quickly.

Oh so its almost like everything else AI related, they basically cheated and lied.<p>If you are shocked by this, you are the sucker in the room.

sometimes o1-preview start hallucinating halfway through a good solution. it can get the intuition and the &#x27;main&#x27; direction for a problem wrong too. but then problem solving is just a series of rephrasings and translating into different math domains is used by mahtematicians for solving problems.

Hmmm without a human control it is not all that clear to me that the variation problems are not more difficult.

There is also a curated benchmark just for those famous problems slightly variated: <a href="https:&#x2F;&#x2F;github.com&#x2F;cpldcpu&#x2F;MisguidedAttention&#x2F;tree&#x2F;main&#x2F;eval">https:&#x2F;&#x2F;github.com&#x2F;cpldcpu&#x2F;MisguidedAttention&#x2F;tree&#x2F;main&#x2F;eval</a>

I think that lamentations about real world data running out is misplaced. We can multiply data with slight variations which might lead to better resilience and more accurate model&#x27;s responses for novel problems.

It still needs to be prompted so it’s easy to understand. If you ask in a weird “how do I not not not win” instead of “ how do I lose” you are gonna run into problems

Yes so when you change the sequence of tokens they&#x27;ve electronically memorized, they get a bit worse at predicting the next token?

If the models are made to pass the benchmarks, of course they’d have some sort of overfit.

i know it&#x27;s easier to just drop the internet tarball on the transformer, but at some point we need to just be giving these models grade school math homework and times tables.

I would love to see how well Deepseek V3 do on this.

Or it&#x27;s time to step back and call it what it is - very good pattern recognition.<p>I mean, that&#x27;s cool... we can get a lot of work done with pattern recognition. Most of the human race never really moves above that level of thinking in the workforce or navigating their daily life, especially if they default to various societally prescribed patterns of getting stuff done (eg. go to college or the military &lt;Based on &lt;these criteria&gt;, find a job &lt;based on the best fit with &lt;this list of desirable skills &amp; experiences&gt;, go to &lt;these places&gt; to find love....)

isn&#x27;t it weird that they didn&#x27;t test gemini?

trained on test. who even trusts OAI anymore?

Yea no shit. LLMs are just REALLY good guessers. People gotta stop the hype lol.<p>Using LLMs for anything serious and which requires consistency and trustworthiness without hallucinations is irresponsible and ridiculous.<p>Closed source LLMs are a bubble and a joke.

MY GOD ITS ALMOST LIKE THEY TRAINED THE MODEL FOR THE BENCHMARK INSTEAD OF FOR GENERAL APTITUDE! WHY WOULD OPENAI DO THIS?!?

Even humans get confused with trick questions right? Once they understand this is a trick question they no longer fall for it. :)

It drops from 50 to 33,96. Still the best, o1 on variable is around 2 times better than Claude on original test.<p>The rest of the llm are far away, single digit.<p>It makes me wonder if o1 is finally getting intelligent? LLM are not supposed to understand these problems when you change variable and values, they have to rely on preexisting data of absolutely identical solved problem to give a correct answer.<p>I didn&#x27;t follow LLM development but I heard one times that chatgpt is now composed of multiple LLM and maybe they put multiple artificial intelligence with purpose of problems solvings or trigonometry for instance.<p>That would explain the reason it&#x27;s so much better.