Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-02 · high
It's a harder, more realistic test than the single-function fixes most benchmarks use for judging legacy-code agents.
Why now: As of July 2, most cited agent benchmarks still score single-function fixes, not the multi-file breaks RepoRescue tests.
RepoRescue tests coding agents by breaking a codebase with dependency and version changes spread across many files at once, per the paper posted on arXiv. That's harder than the single-function fixes most benchmarks score. It's also the test that predicts whether an agent can be trusted on an old codebase, not a fresh one.
Most published agent benchmarks still score single-function fixes: patch one broken call, pass the test, done. RepoRescue skips that setup. It drops agents into a repo where a dependency bump or version change has broken things across many files at once. That's closer to what happens when a legacy project falls behind on updates.
Anyone who's tried to push a coding agent through a multi-file version bump already suspects single-function benchmarks hide the hard part.
Single-function benchmarks are the wrong yardstick for judging whether an agent can handle real legacy work. The model that handles RepoRescue's whole-repo breaks best is the one worth trusting with an actual dependency migration.
As of July 2, most cited agent benchmarks still score single-function fixes, not the multi-file breaks RepoRescue tests.
Each link below shares sources, entities, or timing with this story.
Researchers reviewed public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks: 10.7% were evaluator false negatives rejecting valid alternative solutions, 4.7% were broken or stale tasks. For the genuine failures, verification/feedb...
A June 26 paper (arXiv:2606.26294) describes a self-improving architecture where the agent and the evaluator that scores it evolve together, specifically to avoid the stagnation of optimizing against a fixed, gameable reward. (arXiv) Anyone building a self-improving harness ha...
arXiv 2608.24358 switched models mid-run on long coding tasks using cheap/expensive pairs from the Claude and GPT families. Full-trajectory escalation from weak to strong recovers under half the gap while costing a substantial premium, which the authors call the handoff tax. D...
MemSyco-Bench points out that memory benchmarks test whether memories are correctly stored, retrieved, and updated, never whether the retrieved memory should have influenced the decision at all. Its five tasks check whether agents can reject memory as factual evidence, respect...
arXiv 2607.07989 formalizes failure localization as its own research problem: identifying which agent is responsible and the exact step where the trajectory became irreversibly misdirected. Anyone running a fan-out of research or coding agents has lived this. The output is bad...
An edit cannot un-authorize a permission already granted or un-send a tool request already in flight, and the paper shows an unsafe edit can authorize the same action twice, discard a result the task still needs, or conflict with a call that started before the edit (arXiv 2608...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.