Some critical issues with the SWE-bench dataset

Some of the examples in the paper seem to be wrong.<p>For django-31056, they claim the AI-generated patch is &quot;incomplete&quot; because it&#x27;s &quot;missing critical parts of this logic, such as the try-except block and the check for a running event loop.&quot;. But if you look at the diff, that&#x27;s clearly wrong. The try-except block and running check were <i>already there</i> before the patch. The human patch just indented them, making them appear as both - and +, while the AI patch didn&#x27;t. To me, the AI patch seems correct. It&#x27;s slightly less efficient than the human patch when DJANGO_ALLOW_ASYNC_UNSAFE is set, but slightly <i>more</i> efficient when it isn&#x27;t (which is the common case!). The human patch does feel more natural, but the AI patch is fine. I&#x27;d grade it a tie between human and AI.<p>For django-32517, they claim that the human and AI patches &quot;produce entirely different outputs&quot;, but actually they do exactly the same thing. The human version has `reversed(self.dict)`, while the AI version has `reversed(self.dict.keys())`. `reversed` treats the object as an iterator, and iterating over a dictionary in Python just gives you the keys, so it doesn&#x27;t matter whether you call `.keys()` first. The human patch is more idiomatic, but it&#x27;s also more confusing, as shown by the fact that it confused the authors of this paper. I&#x27;d grade it another tie.<p>Edit: I tried to sign up for OpenReview so I could leave a comment about this, but the system wouldn&#x27;t let me register without completing a form that assumes you have an academic position. Perhaps I should email the authors.

&gt; When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%.<p>This matches my intuition about the coding performance of these models a lot better. I don&#x27;t think any current coding benchmark accurately measures coding performance.

I would argue almost every popular benchmark quoted by the big LLM companies is tainted.<p>OAI, xAI, Antropic, Google all score incredibly well, then you go to try and write code and its just <i>okay</i>.<p>They claim it can do PHD level reasoning, but here I am not trusting it on basic computational thinking.

There’s a few things I’m not understanding here.<p>1. Did the benchmark authors not review the issues and make sure the solution was not present in the issue?<p>2. Are the issues locked after they’re included in the dataset? You’d think they would be immutable for reproducibility.<p>3. For the agents writing patches, is test running part of their inner loop validation? If they write a patch that makes the test pass, then the jobs done. Or is that validation step kept secret from the agent? I don’t see how unless the tests aren’t part of the repo.

Submitted title was &quot;SWE-Bench tainted by answer leakage; real pass rates significantly lower&quot;. Normally we&#x27;d replace that with the article title, in keeping with the site guideline (&quot;<i>Please use the original title, unless it is misleading or linkbait; don&#x27;t editorialize.</i>&quot;), but in this case the article title is so generic that this is arguably misleading as well, so I took a representative phrase from the abstract instead. That&#x27;s preferable, because it&#x27;s better to use the authors&#x27; own representation of their article.<p>If anyone can find a better title (i.e. more accurate and neutral, preferably using language from the article itself) we can change it again.<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;newsguidelines.html">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;newsguidelines.html</a>

So what we need is something like a versioned crowdsourced coding LLM eval dataset.<p>Every quarter, you have a couple thousand volunteers provide 2 GitHub issues from the past 3 months, which are nontrivial to resolve, and where there exists strong test cases. Each volunteer then cross-checks 2 issues from other volunteers. The volunteers get 1 month free subscription to some AI service in return.<p>This dataset is then published as SWE-UberBench-2025-02 or something. People can then only evaluate their coding LLM on datasets published after their training period.

You need benchmarks with the following three properties:<p>1) No known solutions, so there&#x27;s no &quot;ground truth&quot; dataset to train on<p>2) Presumably hard to solve<p>3) But easy to verify a solution if one is provided.<p>This, of course, is easier done on the STEM side of things, but how do you automatically test creativity, or philosophical aptitude?

&gt; 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments.<p>Looking at the benchmark, <a href="https:&#x2F;&#x2F;www.swebench.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.swebench.com&#x2F;</a>, about half of scored submissions score under 1&#x2F;3 correct? So they&#x27;re either not cheating, or not cheating effectively?

The solution moving forward has to be private benchmark suites. I could see teams investing in their own set of programming challenges and periodically re-evaluating them - similar to how we would construct sets of live interview questions for candidates and <i>qualitatively</i> assess their ability.<p>It&#x27;s so vital that it&#x27;s not leaked and that it&#x27;s fit-for-purpose and manually assessed. These general purpose, public benchmarks based on questionable metrics are effectively worthless to assess real programming skill.<p>Case in point, as others have mentioned here, Claude scores modestly on these benchmarks but vastly better than the alternatives in practice. I don&#x27;t trust Claude fully but far more than OpenAI models; it&#x27;s not even close. The IRL performance advantage is not reflected in any of these benchmarks.

My own impression with SoTA models is that they’re very useful for coding, yet they suck ass for solving unique problems (which is the case for every sufficiently large codebase).

There&#x27;s a serious issue with benchmarks.<p>Instead of resolving it, some leaders are further complicating their meaning<p>Such as OpenAI grading their benchmarks based on &quot;how much money they made&quot; or &quot;how easy a model was convinced to hand over fake money&quot;.

I am shocked—<i>shocked</i>—when a vendor cheats in order to increase their benchmark scores.<p>I always tell my customers to ignore benchmarks and compare outcomes with their own workloads. Benchmarks are almost completely useless in the real world.

To quote Goodhart&#x27;s Law: When a measure becomes a target, it ceases to be a good measure.<p>Or, as in the case of LLMs and benchmarks: When a benchmark becomes a target, it ceases to be a good benchmark.

&gt; solutions were directly provided in the issue report or the comments<p>This is fine, many of my real tickets already explain the solution. A good ticket often offers a solution or where to start looking.

I was wondering how long this would take to surface, you can tell a surprising amount just by carefully watching how the trainers answer interview questions, which is kinda meta really.

I found that this paper was submitted to ICLR, but got rejected: <a href="https:&#x2F;&#x2F;openreview.net&#x2F;forum?id=pwIGnH2LHJ" rel="nofollow">https:&#x2F;&#x2F;openreview.net&#x2F;forum?id=pwIGnH2LHJ</a><p>To me the analysis of SWE-Bench is a solid contribution and informative. My guess is that to meet conference&#x27;s submission bar they had to come up with their own bench (SWE-Bench+), which wasn&#x27;t thorough enough and the paper got rejected mainly because of that.

&gt; 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments.<p>Is this what Hofstadter means by a strange-loop?

Something weird (or at least uncommon) that has caught my attention and I havent seen mentioned in the comments is that they cite the swe-bench paper author by first name in the abstract, Carlos et al, and then by last name (as it is usually done) in the paper, Jimenez et al.

Paper from October 2024