A deterministic verifier beats AI monitors on agent failures

A one-class echo-state-network ensemble catches 71% of mid-episode AI agent failures, per a new benchmark of 2,823 episodes across three frameworks. It does that for a fraction of an LLM judge's cost. The gap is the point: judge-based monitoring means calling an LLM every step, while the ensemble runs in about 200 microseconds per step. Cheap enough to watch every action an agent takes.

The catch is that the learned monitor doesn't transfer. Applied cold to a framework it wasn't trained on, its AUROC falls to 0.527, barely better than a coin flip. Recalibrated for that framework, it climbs to 0.885.

A simpler check skips learning entirely. It recomputes a run's stated total from the tool results the agent actually received and flags any mismatch. That catches 60% of failures, eleven points fewer than the ensemble, but produced zero false positives across 1,825 healthy episodes.

Wiring either detector into a rollback-and-rerun loop, killing a flagged episode and retrying it, pushed task success from 52% to 73%. That cost roughly one extra model call per run.

Teams will end up shipping the deterministic verifier over the fancier ensemble. Zero false positives across 1,825 healthy runs matters more in production than catching eleven extra points of failures. The verifier also doesn't need recalibrating for every new framework. The comparison lands as agent frameworks push more autonomy into production without any agreed way to catch a run going bad mid-episode.