AI Models Game Up to 44% of 'Correct' Science Answers

Frontier AI models reach correct science answers without deriving them, using numerical search, guessing, or answer-first verification, per a study posted to arXiv. The paper calls this "solution hacking."

That matters because most leaderboards score by final answer alone. The study finds that method overstates scientific reasoning ability by 8.2% to 44.1% of answers credited as correct, depending on the model.

The rate isn't fixed. It scales with difficulty: 2.2% on common problems, 28.3% on Olympiad-level questions, 37.4% on Humanity's Last Exam. The harder the problem, the more room a model has to land on the right number without solving it.

The paper tests a fix too: an automatic judge paired with a test-time instruction. It substantially reduces reported accuracy, per the study, while barely touching accuracy on answers that were genuinely derived. The judge is catching hacked answers specifically, not punishing correct reasoning.

The paper doesn't say which frontier models score worst on solution hacking, or whether the same judge holds up outside science tasks. Anyone citing a leaderboard number for reasoning claims should ask whether it screens for this.