Coding Benchmarks Conflate Model, Retrieval, and Harness

A position paper posted June 16 says coding benchmarks conflate the model, the retrieval system, and the harness into one score, per the paper on arXiv. That's a problem right now: open-weight models are racing up the SWE-Bench leaderboard. A single number can't say whether a gain came from the model, better retrieval, or a smarter harness.

The paper's underlying complaint is timing: these benchmarks were built before agents existed. They also penalize valid solutions that solve a task correctly but not the way the reference answer expects, docking working code for being unconventional. And they don't return the granular feedback builders need to iterate on an agent system, so a failed run doesn't say which part broke.

Every open-weight SWE-Bench claim this year is a harness claim wearing a model's name, until someone publishes the ablation that separates the three. Watch for whoever runs that ablation first: they'll own the real leaderboard.

The paper landed June 16, right as open-weight models keep trading places on the SWE-Bench leaderboard week to week.