Harness swaps outscored model swaps in AI benchmark

JuliaHub ran four frontier models through five sealed modeling and simulation problems on July 30. The harness gap beat the model gap by more than double: swapping models produced a 0.162-point spread, swapping the harness produced 0.366. For teams building or shipping agents, that means the scaffolding wrapped around a model matters more than which model you pick.

The hardest problem was a full NASA HL-20 flight vehicle with six-degree-of-freedom dynamics. Claude Fable 5 took the top score at 0.889 and swept all twelve trials on the four core problems. GPT-5.6-Sol followed at 0.814, then GPT-5.6-Terra at 0.786, then GPT-5.6-Luna at 0.727.

OpenAI backed this up the same day. Turning on retained reasoning and compaction in the Responses API took GPT-5.6 Sol from 13.3% to 38.3% on the ARC-AGI-3 public set. Output tokens dropped sixfold, same weights.

Under the official evaluation harness, the same model scored as low as 7.8%. Its private chain of thought got discarded after every move, forcing it to rebuild the game state from scratch each turn. Two settings, nearly a 3x swing.

A July 29 arXiv study adds a third data point. Researchers mined 1.1 million Reddit posts across 29 subreddits and isolated 446 threads on security and privacy problems in Cursor, Copilot, and Codex. After reading over 6,000 comments, most reported issues traced to system-level choices, data access scope, unchecked autonomous actions, not the model underneath.

Audit the config before you touch the model

In my own work, every time output quality drops, my first instinct is to blame the model. It's almost never the model. Usually it's a tool grant revoked in a refactor, or a context budget that quietly truncated.

Before you migrate models, check what your setup can actually reach versus what your prompts assume it can reach. Next time you read a leaderboard, ask whose configuration produced the number.