Fetching from the wire…
01
02
03
04
05
06
07
08
Skills2026-06-08 · source-backed
A single passing eval hides nondeterminism. Measure all-runs consistency with pass^k (the probability all k runs pass) so flaky agents get caught before release. Pair it with a judge calibrated against a human gold set, and grow that golden set from real production traces so the metric tracks how your agent actually fails.
Each link below shares sources, entities, or timing with this story.
An agent is dangerous only when it has all three at once: access to private data, exposure to untrusted tokens, and an exfiltration vector. Before shipping, architect to break at least one leg. Strip the outbound channel, sandbox the untrusted input, or scope away the sensitiv...
Here's the single most useful number I've read this week, and it explains nearly every "great demo, broken in production" story I've ever lived. pass@k measures best-case: did at least one of k attempts succeed. pass^k measures consistency: did all k attempts succeed. For an a...
SARC-DQ found competent agents converted freshness/lineage/provenance defects into costly actions about 60% of the time, with both data-quality flags and the agents' own hedging detecting them at chance. The conversion rate was flat across four model tiers spanning a 15x price...
Runta published FrontierHarness on September 2 and it's the most directly useful benchmark I've read this quarter, because it controls the one variable everyone conflates. Nine agent harnesses (Codex, Claude Code, OpenCode, Pi, Oh My Pi, DeepSeek Harness, Kimi Code, Exo Harnes...
Context inconsistency, not pattern choice, is the main reason multi-agent pipelines fail. One agent's malformed output silently corrupts everything downstream. Validate each inter-agent handoff with Pydantic or Zod before passing it on. This single discipline kills most cascad...
Measure four dimensions: consistency (same task, same result), robustness (degraded conditions), predictability (calibrated uncertainty), and safety (bounded error severity). An agent at 90% success with unpredictable 10% failure is an assistant, not an autonomous system. Know...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.