Only 3% of replayed agent states are still valid, so your offline router eval is measuring a world that doesn't exist.

The Replay Gap (arXiv 2608.08239) forked live SWE-bench trajectories at controlled points, rebuilt the environment, and continued each fork with a different model across ~900 rollouts. Model swaps rewrote 61–94% of post-fork actions and diverged at the very first post-fork action 74–77% of the time, versus 6–35% for same-model controls. All five outcome flips occurred in swap arms, zero across 359 control forks. A log-stitching replay evaluator mispredicted every success-relevant outcome and produced patches with 0.00–0.11 similarity to reality. If you route per-step between Haiku and Opus, branch live or don't trust the number.