Public story · 2026-06-30 · high
Confident AI's guide grades agents on trajectory, not answers
Agents can land on the right answer through an unsafe path, and scoring only the output misses it.
Why now: This lands as one specific fix for a known blind spot: scoring answers without ever checking how an agent got there.
Story
Agent-as-judge evaluation checks an AI system's intermediate steps and tool calls, not just its final output, per a guide from Confident AI.
That catches the failure output-only scoring misses: an agent that reaches the right answer through a wrong or unsafe path. In production, that's the run that blindsides a team. It looks clean. It wasn't.
The method runs a cheap agent-as-judge on 100% of traffic: every trace, every tool call, every intermediate step, scored automatically. Teams then pull a small human-reviewed sample, 1-2% of traffic, and check it against the judge's calls. If the two agree, the automated grading holds up.
The 1-2% human sample is the part most teams will cut first. When teams skip it, the judge's blind spots go unchecked: agents start passing on the exact steps a human reviewer would have flagged. Watch for teams that report a judge's pass rate but never publish the human-agreement number. That's the tell they skipped the check.
This lands as one specific fix for a known blind spot: scoring answers without ever checking how an agent got there.
Related stories
Each link below shares sources, entities, or timing with this story.
Shared topic / Tension
OpenAI Responses API Gets Full Computer Environments.
Overlapping topics (agent, call, final, intermediate); pushes against this story (vs).
Shared entity: Output / Earlier coverage / Tension
84% of Developers Use AI Tools. 3% Strongly Trust the Output. Read That Again.
Both cover Output; earlier Output coverage from 2026-04-02; pushes against this story (but).
Shared topic / Tension
MCP tool poisoning fires silently on every call
Overlapping topics (agent, call, catch); pushes against this story (but).
Audit your eval harness against the five-rule checklist.
Overlapping topics (agent, answer, evaluator); pushes against this story (against).
The Red Queen Gödel Machine co-evolves agents with their own evaluators.
Overlapping topics (agent, answer, evaluator); pushes against this story (against).
Shared topic / Downstream implication
Evaluator bias is contagious in multi-agent systems.
Overlapping topics (agent, evaluator, judge); traces where this leads (downstream).
Shared topic / Tension
Gate agent CI on pass^k consistency, not a single green run.
Overlapping topics (actually, agent, human); pushes against this story (against).
Computer-Use Agents Cost 45x More Than APIs. The Numbers Are Brutal.
Overlapping topics (agent, call, human); pushes against this story (against).
Source trail
Entities
Claim evidence
- Confident AI's guide grades agents on trajectory, not answers
Provenance
- Canonical issue
- 2026-06-30
- AI generated
- yes
- Story unit
- 2026-06-30-score-the-trajectory-not-just-the-final-answer
- Labels
- source-backed, canonical briefing excerpt