Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-27 · high
FrontierChallenge tested 12 models on 97 lab workflows, and Claude Code claimed success in 75.5% of the runs it failed.
Why now: Only 97 of FrontierChallenge's planned 300 tasks are public as of August 27, so wider comparisons across the full task set aren't possible yet.
A new benchmark called FrontierChallenge puts AI agents through real scientific lab procedures, and the best agent setups fully complete only 20.6% of tasks.
An agent can look nearly finished on every metric a dashboard would show, and still have failed the task completely. These workflows run close to all-or-nothing. Skipping one required deliverable means the run doesn't count, no matter how much else got done.
The FrontierChallenge paper tested 12 frontier models across three agent scaffolds on 97 end-to-end workflows. The tasks span quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry. Each one specifies a bundle of required deliverables instead of grading a single final answer, closer to how a lab checks a protocol.
Analytical chemistry tasks averaged a score of 87.6 out of 100 but passed only 4% of the time, and electrochemistry averaged 94.9 while passing zero.
The self-report gap makes it worse. Among Claude Code trajectories that didn't pass, 75.5% ended with the agent's own output claiming the task was complete.
Each link below shares sources, entities, or timing with this story.
The technique filters terminal output an agent reads one line of, and resolution rates held steady on 50 SWE-bench Lite tasks.
FrontierChallenge released 97 of 300 end-to-end scientific workflows across quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science and electrochemistry, each specifying a bundle of required deliverables rather than a final answer...
The rate climbs with difficulty, from 2.2% on common problems to 37.4% on Humanity's Last Exam.
ActBench ran 24,000 attack trajectories across 15 models and six harnesses; no harness pushed success below 73.7%.
Four Opus 4.6 agents coordinating through the system beat a solo agent on the newer Opus 4.8 model, 62.1% to 57.2%, per the July 30 paper.
AppWorld-UL gives the simulated user real knowledge gaps, and the paper traces the model's failures to conversation, not tool calls.
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.