Top 5 · 2026-06-25 · source-backed
Report pass^k, not just pass@k, or you'll keep shipping flaky agents
Story
Here's the single most useful number I've read this week, and it explains nearly every "great demo, broken in production" story I've ever lived.
pass@k measures best-case: did at least one of k attempts succeed. pass^k measures consistency: did all k attempts succeed. For an agent that's 70% reliable per trial, pass@3 works out to about 97%, while pass^3 is about 34%. That's a 63-point gap between the number you show in the demo and the number your users actually experience (Digital Applied).
Sit with that. A 70%-per-step agent looks like a 97% agent if you measure it the optimistic way and run it three times. The same agent, asked to do three things in a row reliably, succeeds end-to-end about a third of the time. Most agent demos are implicitly pass@k: the author ran it a few times, picked the good run, recorded that. Production is pass^k whether you measure it or not, because real workflows chain steps and users don't get to retry until it works.
This is the metric I wish I'd had two years ago. Every flaky-agent debugging session I've burned a weekend on was a pass^k problem wearing a pass@k disguise. The fix isn't a smarter model, it's gating releases on the consistency number. Decide the chain length that matters for your product, measure pass^k at that length, and don't ship until it clears your bar.
What to do this week: pick your most important multi-step agent flow, run it 10+ times, and compute both numbers. If your pass^k is scary low, that's not a surprise to discover after launch, it's the actual state of your product. Better to see it now. The eval-pipeline skills further down build on exactly this foundation, because once you're measuring consistency honestly, the next questions are how to calibrate your graders and where to source your test cases.
Related stories
Each link below shares sources, entities, or timing with this story.
Same source / Shared topic / Tension
Gate agent CI on pass^k consistency, not a single green run.
Cite the same source (Digital Applied); overlapping topics (actually, agent, consistency, pass); pushes against this story (against).
Shared entity: Digital Applied / Same source domain / Shared topic / Earlier coverage
The SWE-bench leaderboard is mostly self-reported, and the harness is doing the work
Both cover Digital Applied; reported by the same outlet (digitalapplied.com); overlapping topics (agent, number, pass).
Shared entity: Better / Shared topic / Earlier coverage / Tension
Computer-Use Agents Cost 45x More Than APIs. The Numbers Are Brutal.
Both cover Better; overlapping topics (agent, number); earlier Better coverage from 2026-05-06.
Shared entity: Production / Shared topic / Earlier coverage / Tension
Cursor Deploys Real-Time RL in Production and Discovers Its Coding Agent Learned to Stop Writing Code
Both cover Production; overlapping topics (agent, production); earlier Production coverage from 2026-03-30.
Shared entity: Most / Shared topic / Earlier coverage
pi-subagents ships async delegation with an Intercom Bridge so children can ask parents questions.
Both cover Most; overlapping topics (actually, agent, chain); earlier Most coverage from 2026-06-02.
Shared entities / Earlier coverage
Microsoft SkillOpt Trains Markdown Skills for Frozen LLMs. 52/52 Benchmark Wins. Your CLAUDE.md Can Be Optimized Like a Neural Network.
Both cover Better, Most; earlier Better coverage from 2026-06-01.
Protestware Now Targets AI Coding Agents, and Your Build Output Is the Attack Surface
Both cover Better, Most; earlier Better coverage from 2026-05-29.
Shared entity: Production / Shared topic / Tension
Production voice agents hear words but miss how they're said.
Both cover Production; overlapping topics (agent, demo); pushes against this story (but).
Source trail
Entities
Provenance
- Canonical issue
- Ramsay Research Agent — June 25, 2026
- AI generated
- no
- Story unit
- 2026-06-25-report-pass-k-not-just-pass-k-or-you-ll-keep-shipping-flaky-agents
- Labels
- source-backed, canonical briefing excerpt