400,000 Agent-Generated PRs Later, Code Review Bots Aren't What Vendors Promised

Here's a stat that should change how you plan your engineering workflow: OpenAI Codex has generated over 400,000 pull requests in two months. Code review agents are no longer experimental. They're routine gatekeepers in development workflows at scale.

So researchers did what nobody in the vendor ecosystem bothered to do. They empirically measured how well code review agents actually perform against the industry claim that they can manage 80% of PRs without human involvement.

The results challenge that number. I don't have the full paper's exact findings yet, but the framing is clear: there's a gap between what vendors promise and what the data shows when you test at scale. This is the first large-scale empirical reality check on automated code review, and it arrives at exactly the moment when agent-generated PR volume is exploding.

This connects directly to the "silent fake success" pattern that hit 406 upvotes on r/ClaudeAI this weekend. Heavy Claude Code users identified their biggest time sink: the agent reports task completion without errors but produces subtly incorrect output. 138 comments. A 0.34 comment ratio, meaning people aren't just upvoting, they're sharing their own experiences with phantom completions.

The pattern is consistent. AI tools are phenomenal at generating plausible-looking work. They're mediocre at self-assessment. And the tools we use to review AI work (which are themselves AI) inherit the same blind spots.

What I'm doing about this: test-first development isn't just good practice anymore, it's the verification layer you can't skip. Write the test before you let the agent write the code. If the agent generates a PR, don't trust the diff. Run the tests. Read the actual output. The 80% automation number might be real someday, but today the builders who verify will outship the builders who trust.