SWE-bench Verified is leaking answers, and 60.8% of the "solved" issues prove it

While the capability stories pile up, here's the counterweight. As SWE-bench Verified scores cluster near saturation on July leaderboards, an enhanced analysis (SWE-Bench+, on the AIware 2026 benchmark track) found 60.83% of commonly resolved issues contain solution leakage right in the issue descriptions. Once you strip the leaked hints and the weak tests, model resolution rates drop substantially (AIware 2026).

So a big fraction of what looked like agents reasoning their way to a fix was agents reading the answer off the back of the card. The number cratering when you remove the crib notes tells you how much of the headline was measurement artifact.

This isn't an isolated gripe. It's the honest-measurement thread of the week, and it's got company. GPT-5.6 Sol tops the coding and professional-workflow charts, then lands around 13.3% on public ARC-AGI and 7.8% on the semi-private set even at max reasoning, where humans still hit roughly 100% (ARC Prize). And a sharp arXiv paper, "When the Judge Changes, So Does the Measurement," shows an LLM-as-judge score can move even when the candidate responses are held fixed, purely because you swapped the judge (arXiv). Three different cracks in the measurement layer, same week.

I've been burned by this directly. I picked a coding tool for one of my projects partly on a Verified score and got real-world behavior that didn't match the leaderboard at all. Now I know part of why. The benchmark rewarded reading the issue, not solving it.

What to do, concretely: discount headline Verified numbers when you're choosing between coding agents. Weight contamination-resistant evals like SWE-bench Pro (built on actively maintained repos) and, better, run your own eval on your own real work. Ten tickets from your actual backlog beats a thousand tasks with leaked hints. And if you're doing LLM-as-judge scoring in your eval harness, freeze your judge model. Don't silently upgrade it and assume the scores stay comparable, because the paper says they don't. Benchmark skepticism isn't cynicism here. It's the only way to not get fooled by your own dashboard.