METR Study: Half of SWE-bench Passing AI PRs Would Not Be Merged by Real Maintainers

METR·high signal

METR had 4 active maintainers from scikit-learn, Sphinx, and pytest review 296 AI-generated pull requests that passed SWE-bench Verified automated tests; maintainer merge decisions were on average 24 percentage points lower than automated benchmark scores. Primary rejection reasons were code quality issues, failing edge cases not covered by tests, and not following repository standards. METR notes this is not a fundamental capability ceiling — better prompting and elicitation could close the gap — but it reveals that SWE-bench scores overstate real-world merge readiness.