BenchShield raises reward-hacking detection in agent benchmarks from 23-94% to 77-100% full-chain recall, using a lifecycle model of reward events

arXiv·medium signalXBlueskyLinkedInCopy link

arXiv 2609.11028 models the reward-relevant events of an agent benchmark as a finite lifecycle. It runs static taint analysis on that model to expose hacking paths before a run, and uses evidence from the benchmark infrastructure to attribute agent actions during it. On 456 human-adjudicated trajectories mined from more than 31,000 public runs across three benchmarks, it beats an agentic hackability scanner and cuts per-task cost by up to 65%. Its runtime detection is 96% accurate. For builders reading leaderboard numbers, it's a concrete way to check whether a score came from solving the task or gaming the grader.