A Small RL-Trained Model Diagnoses Agent Drift

A team trained a small language model to diagnose and recover from agent drift. The approach skips retraining the larger model that's already running, per a paper posted to arXiv. The system routes each failure through a four-stop recovery graph. It classifies the drift, flags the operation that caused it, weighs the risk, and decides how to recover.

Retraining a production agent's main model every time it wanders off course is expensive and slow. A specialized side model only needs four narrow judgment calls right, which makes it cheaper to iterate on than the deployed agent.

The reward signal for that side model mixes two checks. A rule-based check confirms the output matches the expected XML schema. A separate LLM acts as judge, scoring whether the recovery decision itself is correct, not just well-formatted.

Tested on the AppWorld benchmark, the trained model held its schema and made correct recovery calls at each stage, per the paper. The paper doesn't say how the approach holds up outside AppWorld, or what happens when drift compounds across multiple agent runs.

Recovery sidecars, not retrained main models, will become the default fix for agent drift in production. A small RL-trained model is far cheaper to retrain than the model actually doing the work. Cheap retraining wins once teams run agents past a single demo.