An Auto-Optimized Agent Harness Can Cheat the Whole Benchmark, Not Just the Tasks
arXiv 2609.18366 (submitted 16 Sep 2026) identifies a hole in harness optimization: a Proposer that repeatedly edits prompts, memory, retrieval, tools and control code against a released benchmark can find a benchmark-wide protocol shortcut, which task holdout will not catch because holdout varies semantic tasks while leaving the protocol fixed. CHASE recasts harness evolution as constraint generation over validity-preserving benchmark counterfactuals: after each Proposer update, a Challenger searches for an executable protocol transformation that destroys the gain, a validity firewall checks task semantics are preserved, and a confirmation set decides. For anyone tuning a harness against SWE-style benchmarks, this says holding out tasks is not holding out anything.
↳ Follow the thread