Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-25 · high
A three-stage check across 520 agent runs found language rewrites scoring 5.6, far below build-toolchain rewrites' 31.4.
Why now: SWE Refactor Bench's numbers are new to coverage as of August 25, with no separate release date given in the paper itself.
A new benchmark tested coding agents on 20 whole-repo stack migrations, and only 28 of 520 runs cleared every check. It's called SWE Refactor Bench. Detailed in a paper posted to arXiv, it grades each run through a migration audit, behavioral tests, and an independent verification agent.
Thirteen of the 20 tasks got no accepted solution from any model, and the best performer, claude-opus-5, scored 47 out of 100.
Build-toolchain rewrites scored 31.4 out of 100. Language rewrites scored lower, at 5.6.
The paper names the dominant failure mode Blindness. Of the 340 runs that cleared the audit stage, 58% reached 99% of fixed checks, but only 26% reached 100%. That last point is where most runs die, invisible to a test count that only shows how close a run looks, not whether it's done.
I've seen this pattern with my own agent use. A refactor's tests go green, and then a second pass catches something the tests missed. This benchmark puts a number on how often that gap causes a failed run.
Each link below shares sources, entities, or timing with this story.
SWE-NFI builds 188 tasks from merged Python PRs and operationalizes non-functional improvement as 92 executable rules, cleanly separating "tests still pass" from "the code got better." Best agent: 70.0% functional correctness, 0.0-1.3 on structural improvement against a human...
The PlayCoder benchmark is the cold shower the vibe coding movement needed. Researchers tested 10 state-of-the-art code LLMs on generating GUI applications across six categories. The models achieved high compilation rates. The code built and ran. But when they measured whether...
Thinkingbox is an MCP-compatible sandbox with isolated sessions, full execution traces, and outcome evaluation against terminal backend state, carrying 507 policy-conditioned workflows across retail, hospitality, auto insurance, neobank internal IT and consulting support (arXi...
VAKRA (arXiv 2608.12282) benchmarks agents against 8,000+ executable APIs across 62 domains, verifying by re-executing predicted calls against live endpoints. Accuracy falls to 50-51% on compositional APIs and degrades over 50% as depth grows. Failures concentrate in entity di...
157 tasks from 27 repos. Best resolved rate: 29.94%. Agents cause scope creep and regressions by diverging from user intent. Practical implication: agents need explicit scope constraints. arXiv 2509.22237 ---
arXiv 2608.09802, accepted at COLM 2026, audits the benchmark everyone quotes in funding decks and finds a large chunk of the reported headroom is measurement error, not model failure. Their replacement is 170 expert-curated multilingual refactoring instances across Python, Ja...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.