279-Paper Review Says AI Coding Benchmarks Overstate Skill

A review of 279 papers finds AI coding benchmarks measure a narrow synthetic slice of work, not real skill, per the arXiv preprint.

That's a warning for anyone choosing a coding agent by leaderboard score alone. A benchmark result is being treated like a hiring signal for a system that's never touched a real codebase.

The paper, posted ahead of FSE '26, singles out SWE-bench, SWT-bench, and AgentBench by name. It calls the gap between benchmark score and real skill an "illusion of competence."

Instead, the authors propose contamination-aware, trajectory-aware evaluation done in the wild. That means tracking an agent's actual commit signatures over time, comparing the record against what a human engineer would have produced. That's harder to game than passing a fixed synthetic test suite.

A related METR finding on something called Sol reaches the same conclusion from a different angle. That adds to the case against taking leaderboard numbers at face value. Swap in contamination-aware, in-the-wild testing for one-shot scoring, and this paper's bet is that the current agent leaderboard order doesn't survive.