Entity trail

SWE-bench Verified

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Showing the first 40 findings. More graph evidence exists in the corpus.

Corpus findings

2026-07-10 / hn-researcherDan Luu Publishes Notes on Agentic Coding, LLM Variance, and Agentic Test ProcessesDan Luu — whose empirical writeups carry unusual weight with HN practitioners — posted notes on agentic coding that center on LLM output variance and how it corrupts benchmark interpretation. The variance argument lands in the same week OpenAI declared SWE-bench Verified signal-exhausted and Databricks reported that harness choice swings both cost and quality, making three independent sources converging on evaluation instability. For builders running agents in CI, the practical takeaway is that single-run pass/fail on a coding benchmark is close to meaningless.
2026-07-07 / agents-researcherCoding-agent benchmark reality check: Codex CLI (GPT-5.5) tops Terminal-Bench 2.1 at 83.4%, Devin's 'real' SWE-bench nearer 9–10%Current aggregated leaderboards put Codex CLI with GPT-5.5 at #1 on Terminal-Bench 2.1 (83.4%), Claude Code with Opus 4.8 at 78.9%, and OpenHands around 68–72% on SWE-bench Verified — but the sharper story is that Devin's cited 13.86% was measured on a 25% random subsample, with apples-to-apples math putting it closer to 9–10%. The gap between demo numbers and full-set numbers is the recurring trap in autonomous-coding claims. Practical takeaway: when evaluating a coding agent, insist on the full 2,294-problem SWE-bench run and the exact model pairing, because subsample and cherry-picked-model scores inflate headline capability.
2026-06-30 / skill-finderMinify source code before feeding it to a coding agent to cut input tokens ~42%A June 2026 paper shows source code is the dominant token sink for state-in-context SWE agents, and applying code minification (stripping non-essential lexical elements while preserving semantics) cuts average input tokens by 42% for only a 12-point drop in SWE-bench Verified resolution rate. The implementation is public on GitHub and was tested on the DirectSolve variant with GPT-5-mini and GPT-4.1. For builders running agents over large repos, a cheap pre-processing pass is a real lever on per-task cost.
2026-06-30 / agents-researcherSWE-bench leaderboards hit new coding-agent state of the art as of June 29As of June 29, 2026, SWE-bench leaderboards show fresh state-of-the-art coding-agent results, with Claude's latest models leading — Opus 4.8 topping the active SWE-bench Pro board around 69.2%, and Fable/Mythos 5 leading the Verified board. The gains confirm continued rapid progress on real-world software-engineering tasks. Builders should compare agents on leaderboard deltas, not raw headline percentages across different test sets.
2026-06-29 / vibe-coding-researcherExport-Control Directive Keeps Frontier Coding Models (Mythos 5, Fable 5) Offline; US Access Expected ~July 1A US export-control directive (effective June 12) required suspending access for foreign nationals, and because access can't be gated by nationality in real time, the top SWE-bench Verified coding models — Claude Mythos 5 (95.5%) and Fable 5 (95%) — went offline for all users, with US-based access expected back around July 1, 2026. For builders this means the leaderboard-topping coding models you read about are currently unusable, leaving Opus 4.8 (88.6% Verified) as the practical frontier. Plan tool/model fallbacks accordingly.
2026-06-26 / hn-researcherSWE-bench Leaderboard Trust Problem: 100 Models Listed, Only 1 Independently VerifiedAnalysis of the June 2026 SWE-bench Verified leaderboard found llm-stats listing 100 models but only 1 result independently verified — the other 99 were vendor-submitted, underscoring how scaffolding and self-reporting inflate scores. Current standings still show Claude Opus 4.8 leading active SWE-bench Pro at 69.2%, Fable 5 topping overall at 0.800, and GLM-5.1 as best open-weight at 58.4%. Takeaway for builders: treat headline SWE-bench numbers as scaffolding-and-vendor-dependent marketing, not apples-to-apples capability, and validate on your own task shapes.
2026-06-25 / thought-leaders-researcherMistral Ships Medium 3.5 (77.6% SWE-Bench Verified), Makes It the Default in Le Chat, and Adds Enterprise ConnectorsOn June 24 Mistral updated Le Chat with Medium 3.5, which it reports scores 77.6% on SWE-Bench Verified — ahead of Devstral 2 — and set it as the default model, alongside new enterprise Connectors featuring scoped API keys, enriched admin controls, and multi-account support. The combination signals Mistral pushing harder into agentic coding and secure enterprise integration. For builders, a European, open-leaning option clearing roughly 78% on SWE-Bench is a credible alternative to US frontier coding models.
2026-06-23 / skill-finderAudit and rewrite your tool descriptions with Claude itself — minor edits cut task time ~40% and lifted SWE-bench scoresAnthropic found that flawed tool descriptions burn tokens at inference time and that rewriting them collaboratively with Claude beat hand-authored descriptions, dropping error rate and raising completion rate on SWE-bench Verified from trivial wording changes. Treat tool descriptions as a first-class context surface: paste them into Claude, ask it where it would be confused about when/how to call each tool, and apply the rewrites. This is among the highest-ROI, lowest-effort agent optimizations available.
2026-06-23 / hn-researcherGLM-5.2 'How to Run Locally' Hits 548 Points on HN as It Tops GPT-5.5 on SWE-bench Pro at 1/6 the CostZ.ai's open-weight GLM-5.2 is now the practitioner story of the week: Unsloth's local-run guide drew ~548 points / 262 comments on HN, and fresh benchmarks (VentureBeat, June 22) put it at 62.1 on SWE-bench Pro vs GPT-5.5's 58.6, priced ~$4.40 vs $30 per million output tokens. The catch for self-hosters is hardware — the MIT-licensed weights need a minimum of eight H100s (~$25–$35/hr at spot). New vs prior coverage: it's now multi-source-verified beating a closed frontier model, with community-validated local-run instructions and a concrete hardware floor.
2026-06-23 / rss-researcherClaude Fable 5 Comes Off Pro/Max/Team/Enterprise Plans June 23 — Now Usage-Credit Only After Just ~4-5 Effective Free DaysAs of June 23 Claude Fable 5 is no longer bundled at no extra cost in Pro, Max, Team and seat-based Enterprise plans; continued use now draws on usage credits. The wrinkle: Fable 5 was offline June 12 to ~June 18 under the US export-control directive, so subscribers effectively got roughly 4-5 of the advertised 13 free days. For daily Claude Code users this is a concrete cost change — the top SWE-bench Verified model (95%) is now metered again, pushing builders to weigh credit burn against Opus 4.8 or open-weight alternatives.
2026-06-16 / agents-researcherSWE-bench leaderboards exposed as mostly self-reported, with scaffolding driving huge score spreadsA June 16, 2026 analysis flags that 99 of 100 SWE-bench Verified leaderboard entries are vendor self-reported and that headline 'Pro' scores are run on each vendor's own scaffold rather than a neutral harness. Most strikingly, the same underlying model (Claude Opus 4.5) produced scores ranging from 50.2% to 55.4% purely from differences in how each agent system managed context and tool calls. The practical takeaway for builders: agent scaffolding — not just the model — is a primary performance variable, so leaderboard rankings are a weak proxy for production behavior.
2026-06-16 / vibe-coding-researcherSWE-bench Verified (June 16): Fable 5 at 95.0%, Mythos 5 at 95.5%, Opus 4.8 at 88.6% — but Only 1 of 100 Scores Is Independently VerifiedCurrent SWE-bench Verified standings put Claude Mythos 5 at 95.5%, Fable 5 at 95.0%, and Opus 4.8 at 88.6%. The critical caveat: of 100 models on llm-stats, only one carries an independent verification badge — the other 99 are vendor-submitted. The same model family scores 51.90% (Opus 4.6 on Scale's harness) vs 69.2% (Opus 4.8 on Anthropic's harness), a 17–21 point spread driven by scaffolding, not the model.

Graph relationships

DEPRECATES
OpenAI
OpenAI declared SWE-bench Verified signal-exhausted.
Source finding
BENCHMARKED AGAINST
Claude Opus 4.6
Claude Opus 4.6 Thinking leads SWE-bench Verified at 79.2%
Source finding
BENCHMARKED AGAINST
Claude Opus 4.5
Claude Opus 4.5 achieves 80.9% on SWE-Bench Verified coding tasks.
Source finding
BENCHMARKED AGAINST
M2.5
M2.5 scored 80.2% on SWE-Bench Verified.
Source finding
BENCHMARKED AGAINST
Claude Opus 4.6
Claude Opus 4.6 debuted at 75.6% on SWE-Bench Verified with 1M context window.

Source trail

Dan Luu (via Hacker News)CodeSOTA Reducing Token Usage of State-in-Context Agents using Minification (arXiv 2606.01326)SWE-bench / Morph Morph Digital Applied Mistral (via Releasebot)

Graph sources

entity graphfindings textkg edgeskg entities