Auto-promote failing production traces into your offline eval set so the test suite grows itself

Digital Applied·medium signal

Share one data layer between production traces and offline eval datasets so any trace that fails an online scorer is automatically promoted into the offline eval set — the suite grows from real user behavior with no manual export. This matters more as tasks lengthen: Anthropic reports the 99.9th-percentile turn duration nearly doubled from Oct 2025 to Jan 2026, so static eval sets drift out of distribution fast. The pattern turns evaluation from a one-time artifact into a self-maintaining feedback loop.