Self-repair needs a failing test, not just an error

Small, frozen code models fix their own bugs only when they're shown a failing test, not just the error, per a preregistered, placebo-controlled study on arXiv. The study ran real self-repair attempts against a placebo arm where models just saw their broken code again, no test, no assertion. The placebo did nothing. The gap is the whole finding: it's the failing test that carries the signal, not another look at the mistake.

That matters for anyone running an agentic loop where a model retries after failing. If your harness just hands the model its own error message and says 'try again,' you're skipping the part that actually works. The study frames it as re-exposure versus falsification: showing a model what it got wrong doesn't help much. Showing it a concrete, executable counterexample does.

The paper doesn't say how big the effect is in absolute terms, only that falsification is the mechanism doing the work. It also only tests small, frozen models, so it's an open question whether the same gap holds for larger models, or ones that get fine-tuned mid-loop.

Any self-repair pipeline that retries on 'here's your error' instead of 'here's the assertion you violated' is leaving fixes on the table, and that's a testable claim, not a guess. It lands as I lean harder on Claude Code to fix its own failing tests in my own projects. If you're building agent loops, wire in the oracle, not the stack trace.