SWE-bench Verified Under Scrutiny: 60.8% of Commonly Resolved Issues Show Solution Leakage

AIware 2026 (SWE-Bench+)·medium signal

As SWE-bench Verified scores cluster near saturation on July 2026 leaderboards, an enhanced benchmark analysis (SWE-Bench+, AIware 2026 benchmark track) finds 60.83% of commonly resolved issues contain solution leakage in their issue descriptions, with model resolution rates dropping substantially once leaked hints and weak tests are removed. The finding reinforces a growing 'SWE-bench is not enough' consensus and the shift toward contamination-resistant evals like SWE-bench Pro on actively maintained repos. Builders comparing coding agents should discount headline Verified numbers and weight Pro/real-work evaluations instead.