SWE-bench leaderboards exposed as mostly self-reported, with scaffolding driving huge score spreads

Digital Applied·medium signal

A June 16, 2026 analysis flags that 99 of 100 SWE-bench Verified leaderboard entries are vendor self-reported and that headline 'Pro' scores are run on each vendor's own scaffold rather than a neutral harness. Most strikingly, the same underlying model (Claude Opus 4.5) produced scores ranging from 50.2% to 55.4% purely from differences in how each agent system managed context and tool calls. The practical takeaway for builders: agent scaffolding — not just the model — is a primary performance variable, so leaderboard rankings are a weak proxy for production behavior.