SWE-bench Is Contaminated. The Primary Coding Agent Benchmark Is Broken.

OpenAI stopped reporting SWE-bench Verified scores. The reason: every frontier model has been trained on the dataset. Morph LLM published the numbers that explain why. Claude Mythos Preview scores 93.9% on the contaminated Verified benchmark. On the new, uncontaminated SWE-bench Pro, the best score is 57% (GPT-5.3-Codex). Claude Opus 4.5 under standardized scaffolding (SEAL) hits 45.9%.

That's a 35-point gap. Let that sink in. The benchmark that every coding agent company put in their marketing decks, the one Devin and Cursor and every agent startup cited to prove their tool actually works, was inflated by contamination. A model scoring 93.9% on Verified doesn't mean it solves 94% of real coding problems. It means it memorized the test.

This connects directly to the APEX-Agents-AA benchmark from Artificial Analysis, which launched this week. APEX evaluates agents on 452 real professional tasks spanning investment banking, consulting, and corporate law with 5-10 day simulated engagements. The top score? GPT-5.4 at 33.3%. Claude Opus 4.6 at 33.0%. Models that claim 90%+ on coding benchmarks fail two-thirds of professional workflow tasks.

The Stanford 2026 AI Index released the same week adds another dimension: "For complex, interactive technologies such as AI agents and robots, benchmarks barely exist yet." We've been evaluating agents with broken rulers and wondering why production results disappoint.

What to do about it: Stop citing SWE-bench Verified scores when evaluating coding agents. Use SWE-bench Pro and APEX-Agents-AA instead. If a vendor won't share Pro scores, that tells you something. And recalibrate your expectations: the honest state of the art for coding agents on uncontaminated tasks is around 57%, not 94%.