Reinforcement-learning reward shrinks test suites, lifts mutation scores

A reinforcement-learning method for generating unit tests, called Ockhamareto, reaches a 49.9% mutation score using an average of 2.6 tests per function. A standard RL baseline needs 4.7 tests to reach 31.3%.

Teams that build coding agents to write test suites often hardcode a fixed test count into their pipelines. The paper checked whether cheap signals, function size among them, could predict where more tests stop paying off. None did.

Ockhamareto's reward rule only reinforces a test rollout when it sits on the Pareto frontier for two measures at once. Those measures are mutants killed and tests spent killing them. It also traces each test's killing power back to the specific source tokens it covers. Against MIST-RL, an existing reinforcement-learning baseline for the same task, that scoring produces a 3.4x better per-test trade-off. Mutation scores rise 30 to 35 percentage points across 4B, 9B and 27B parameter models, on HumanEval+, MBPP+, CodeContests and TestGenEval-Lite.

There's no shortcut here. The paper found no cheap proxy, not even function size, that predicts the efficient test count in advance. A CI policy that caps generated tests at a fixed number per function is guessing, not measuring. Watch whether test-generating agents start exposing a mutation-score target instead of a hardcoded count.