Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-31 · high
The errors trace back to how the benchmark pairs pull requests with GitHub issues, not just to model quality.
Why now: The audit landed on arXiv July 30 and is fresh enough that most people citing SWE-bench scores haven't seen it yet.
PAIChecker, a new multi-agent auditor, found that 13.6% of SWE-bench Verified instances have problem statements that don't match their solution patches, per the July 30 arXiv paper.
SWE-bench Verified is a widely cited coding benchmark, and vendors cite score gaps of a few points as evidence of progress. A 13.6% ground-truth error rate makes deltas that small impossible to trust on their own.
The misalignment traces back to how the benchmark gets built: SWE-bench Verified pairs a real pull request with whatever GitHub issue its description references. It then treats the issue text as the problem statement and the merged patch as the answer key. Nothing in that process checks whether the PR actually fixes the issue it's linked to.
PAIChecker itself is a multi-agent system that combines pattern identification, cross-agent label synthesis, and code-level validation across five misalignment patterns and eleven sub-scenarios. It hit 92.12% binary accuracy on SWE-Gym and 91.67% on SWE-bench Multilingual, tested across four different LLM backbones. That range suggests the detection method generalizes rather than overfitting to one benchmark's quirks.
It joins other new benchmark audits, including separate checks on pentest-agent decoys and COBOL-migration validation.
Each link below shares sources, entities, or timing with this story.
Cross-vendor AI review still shows up in just 1.6% of agent-authored pull requests, but reviewers grade outside code more harshly than their own.
It automates the data-flow, crash-semantics, and commit-history work engineers do by hand.
A blinded judge checks root cause and impact against 95 real CVEs, and no frontier model made the ten-model lineup.
Testing eight models across 192,000 evaluations, researchers found chain-of-thought and direct instructions to ignore the score didn't remove the bias.
The rule text can survive context compaction while the behavior it enforces quietly stops, and grepping the summary for that text won't catch the difference.
A training-free fix called ChannelGuard held steady across three model backends, filter or no filter, blocking every tool-poisoning attempt.
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.