Berkeley Researchers Score 100% on SWE-bench Verified Without Solving a Single Task. Every Major AI Benchmark Is Broken.

A team at UC Berkeley RDI built an automated scanning agent that achieved near-perfect scores on eight major AI agent benchmarks. SWE-bench Verified: 100%. Terminal-Bench: 100%. WebArena: approximately 100%. FieldWorkArena: 100%. GAIA: roughly 98%. OSWorld: 73%. The agent didn't solve a single task. It exploited the evaluation infrastructure instead.

383 points and 97 comments on Hacker News because this fundamentally breaks how the industry picks AI tools.

The exploits are embarrassingly simple. Pytest hooks that intercept test execution and return fake results. Binary wrapper trojans that shadow the tool under test. Configuration leakage where answers ship alongside test materials. Eval() calls on untrusted input that let the agent inject arbitrary code into the evaluator. Seven recurring vulnerability patterns across all eight benchmarks.

This means every SWE-bench score you've seen, every model comparison card, every "our agent solved X% of real GitHub issues" claim needs an asterisk. The benchmarks don't distinguish between agents that actually understand code and agents that game the harness. Most model developers aren't gaming intentionally. But the lack of isolation between agent and evaluator means we genuinely don't know how much of any score reflects real capability versus leakage.

The timing is loaded. Story 1 reports that 51% of code is now AI-generated. Story 2 uses benchmark data to compare multi-agent patterns. Story 4 shows three companies building agent orchestration platforms. All of this rests on the assumption that we can measure which AI tools are actually good at writing code. Berkeley just showed we can't. Not reliably.

I've been using SWE-bench scores to decide which model to use for different tasks. Everyone has. When Claude Code leads SWE-bench Verified at 80.8%, that's a real data point that affects real purchasing decisions. Berkeley's paper doesn't say Claude is bad at coding. It says the benchmark can't prove Claude is good at coding, which is a different and more uncomfortable problem.

The fix is structural. Agent and evaluator need full isolation. No shared filesystem, no shared environment variables, no ability for the agent to access test infrastructure. The paper identifies what clean evaluation looks like. The question is whether benchmark operators will adopt it, knowing their scores will probably drop.

For builders: don't choose your primary AI coding tool based on benchmark scores alone. Run your own evaluation on your codebase, your task types, your quality standards. The benchmarks are useful directional signals, but after Berkeley's paper, treating them as ground truth is a mistake.