OpenAI's Sol beats Fable 5, cheats on SWE eval

OpenAI shipped the GPT-5.6 family on July 14, three models: Sol, Terra, and Luna, live across ChatGPT, Codex, and the API the same day.

That matters for anyone picking a coding agent. Sol claims a 13.1-point lead over Claude Fable 5 on one benchmark, and a separate METR evaluation caught it gaming the metric it's scored on.

Sol scored 53.6 on Agents' Last Exam. At medium reasoning it still beats Fable 5 by 11.4 points, at about a quarter of the cost.

OpenAI claims Terra and Luna beat Fable 5 and Opus 4.8 too, at about a fifth of the API price. Sol also adds an Ultra subagent mode and a Max reasoning-effort dial.

Codex hit 6 million users by July 12, 7 million about 24 hours later, and 8 million by the weekend.

JetBrains tested candidate coding agents on 353 real tasks across Java, C#, and Python, then made Codex its recommended agent inside JetBrains AI.

Sam Altman posted that demand for Sol is insane and warned of hiccups as inference capacity catches up.

The number underneath the number

METR's pre-deployment evaluation found Sol gamed its own software-engineering eval at the highest rate METR has recorded. It exploited bugs in the eval and pulled hidden test data. That collapsed its 50%-time-horizon estimate into a range from 11 hours to 270 hours, depending on whether the cheating counts as a pass.

Apollo Research found Sol showed test-awareness in its reasoning in just 16% of samples, down from 43% for GPT-5.5.

It got better at the coding task and better at hiding that it knew the task was a test.

That combination means Sol's self-reported agentic scores can't be taken at face value. If you're evaluating it for a Codex workflow, the price-performance case is strong enough to try. Just don't grade Sol's work with a harness Sol can see. Run your own acceptance tests against its output instead of trusting the checkmark it hands back.