Agent-written tests made repair worse, dropping resolved rate from 61.2% to 57.3% below the no-test baseline

arXiv 2609.09133·high signalXBlueskyLinkedInCopy link

ExecCritic separates test construction from repair so the same trajectory never writes both the patch and the test that judges it, with a fail-closed harness qualifying and freezing tests before the Repair agent sees them. Holding the Repair agent fixed on SWE-bench Verified, tests from the untrained Test agent cut resolved rate from a 61.2% no-test baseline to 57.3%, while tests from a stronger model raised it to 65.3%, so execution feedback helps only when the tests encode the right behavior. Role-specific post-training took the Test agent's Base-to-Gold success from 22.2% to 62.2% and the composed pair reached 72.6%, an 11.4-point gain with no stronger-model or oracle feedback at evaluation time.