Mid-generation handoff beats routing at equal cost

A new stopping rule lets an AI model decide mid-generation whether to escalate to a bigger model, according to a paper posted to arXiv. Most cascades pick the model before generation starts and stick with it. This one lets a small model start answering, track its own confidence as it goes, and hand off partway through if it's headed for trouble.

For anyone running a cheap-model-first setup, that's the gap between paying for the big model on every uncertain call and paying for it only when the small model needs help.

The method treats the handoff as an optimal-stopping problem. It builds a competence posterior from labelled trajectories rather than raw entropy, derives a closed-form myopic threshold for the decision, and proves the optimal policy is a threshold that shifts over the course of generation instead of staying fixed. The authors also derive a regret bound that shrinks with the square root of the sample size.

They tested it on a Qwen2.5-Coder cascade, small model at 1.5B parameters and large model at 7B, across 257 MBPP coding tasks. Two of three pre-registered predictions held up, including that the escalation frontier dominates post-hoc routing at matched cost. The paper doesn't say which prediction missed.

What stands out is the pre-registration itself. Locking in predictions before running the cascade is rare in agent research, and having two of three hold up gives the result more weight than a paper that picked its claims after seeing the numbers.