Cognition built a coding benchmark that real maintainers grade, and the best model scores 13.4%

For about a year the ambient message has been "coding is mostly solved." SWE-Bench numbers crept past 50%, vendors put them on slides, and a lot of people quietly concluded the hard part was over. Cognition just dropped a bucket of cold water on that.

On June 8 they launched FrontierCode, a benchmark built by 20+ open-source maintainers who each spent 40+ hours constructing tasks. The point isn't "did the test pass." It's "would this PR actually merge." They grade across correctness, tests, scope, style, and maintainability. The numbers are humbling. Claude Opus 4.8 leads the Diamond subset at 13.4%. GPT-5.5 hits 6.3%, Opus 4.7 5.2%, Gemini 3.1 Pro 4.7%. On the easier Main and Extended tiers Opus reaches 34.3% and 51.8%. So far below the SWE-Bench regime that Cognition flatly says real-world coding is "much less solved" than the leaderboards imply. They also claim 81% lower false-positive rates than SWE-Bench Pro, and they're refusing to publicly release the tasks to avoid contamination.

Why this matters: it lines up exactly with what I feel every day. The agent writes code that runs. Whether it's code I'd merge into something I have to maintain for two years is a totally different question, and that gap is where my actual time goes now. FrontierCode is the first benchmark I've seen that measures the gap instead of pretending it doesn't exist. It pairs cleanly with the "Cleaning Up After AI Rockstar Developers" essay that hit HN the same day, arguing agent output ships fast and leaves maintenance debt for teammates to absorb. Same observation from two directions.

What builders should do: stop quoting SWE-Bench at people. If you're evaluating a coding agent for your team, build a tiny private eval out of your own real PRs and grade on mergeability, not green checkmarks. And budget review time honestly. The velocity is real, but the review tax is the new bottleneck, and it doesn't show up in the demo.