CAFE trains a search agent and critic as one model

CAFE trains one model to act as both a search agent and its own critic, alternating between the two roles inside a single training run.

That matters for anyone building agentic search. CAFE's ablation found gains plateau when only the agent or only the critic improves, and climb only when both train together across all seven benchmarks tested.

The corrective feedback isn't automatic. The agent has to request it mid-task, and online reinforcement learning shapes when to grant that request. The training signal comes from the gap between the agent's success rate when it calls for help and when it skips it.

A separate offline stage teaches the feedback itself, trained through preference optimization on matched pairs of successful and unsuccessful trajectories. Online RL decides when to step in. This stage decides what the intervention says.

Across those seven benchmarks, CAFE outperformed the RL-trained baselines it was tested against, and kept its lead on six additional benchmarks it wasn't tuned for.