DRACO Redistributes One Rubric Score Across Agent Steps and Beats Ground-Truth-Reward GRPO by 5.3 Points on AppWorld

arXiv 2609.04094·medium signalXBlueskyLinkedInCopy link

RL from verifiable rewards works when a task has a programmatic checker, but most long-horizon agent domains have none, and the usual fallback of a multi-criteria rubric scored once per trajectory gives one scalar across tens of steps. DRACO generates rubrics dynamically during training to track the policy's evolving capability, scores them once per completed trajectory, then redistributes that judgment in closed form over the steps responsible for annotated rubrics to produce per-step GRPO advantages, with no trained attribution module. It gains 15.9 points over the base model on AppWorld and 5.3 points over GRPO trained on a sparse ground-truth reward despite using no verifier, and 5.3 points on out-of-domain Tau-Bench without a frontier judge; code is released.