Public story · 2026-02-24 · source-backed
SWE-bench February Analysis: The 80% Ceiling and What It Means
Story
simonwillison.net — Four models now above 80% SWE-bench (Opus 4.6 at 80.8%, Gemini 3.1 Pro at 80.6%, GPT-5.3-Codex at 80.0%, Kimi K2.5 at 76.8%). Willison's analysis: we're approaching the ceiling where SWE-bench stops being a meaningful differentiator. The gap between models is now smaller than the gap between scaffolding approaches. This is the quantitative backing for the "scaffolding > model" thesis.
Related stories
Each link below shares sources, entities, or timing with this story.
Shared entities / Shared topic
LogRocket February Power Rankings: Open-Source Breaks Into Top 5
Both cover Codex, Gemini, GPT, Kimi K2; overlapping topics (kimi, model, swe-bench).
Shared entities / Same source domain / Shared topic / What happened next / Tension
GPT-5.5 Is Here. It's Fast, It's Expensive, and It Changes How You Prompt.
Both cover Codex, GPT, Opus, SWE; reported by the same outlet (simonwillison.net); overlapping topics (gpt-5, model).
GPT-5.4 Mini ($0.75/M) and Nano ($0.20/M) — Purpose-Built for Subagent Workloads
Both cover Gemini, GPT, Opus, SWE; reported by the same outlet (simonwillison.net); overlapping topics (gpt-5, model).
Shared entities / Shared topic / What happened next / Tension
Kimi K2.6: A 1-Trillion-Parameter Open-Weights Model Just Beat GPT-5.4 on SWE-Bench Pro
Both cover GPT, Kimi K2, Opus, SWE; overlapping topics (gpt-5, model, swe-bench); picks up the GPT thread on 2026-04-21.
Shared entities / Shared topic / What happened next
June 2026 was the biggest model launch month ever, and ChatGPT just dropped below 50% share
Both cover Gemini, GPT, Kimi K2, Opus; overlapping topics (gemini, gpt-5, model); picks up the Gemini thread on 2026-06-21.
Shared entities / Shared topic / What happened next / Tension
Kimi K2.6 Beats GPT-5.4 on SWE-Bench Pro. It's Open-Weight. It Costs 88% Less.
Both cover GPT, Kimi K2, SWE; overlapping topics (gpt-5, kimi, model, swe-bench); picks up the GPT thread on 2026-05-05.
Shared entities / Same source domain / Shared topic / What happened next
GLM-5.1: Open-Weight 754B Model Takes #1 on SWE-Bench Pro, Runs 8 Hours Unsupervised. MIT Licensed.
Both cover Gemini, GPT, SWE; reported by the same outlet (simonwillison.net); overlapping topics (above, model, swe-bench).
Shared entities / Shared topic / What happened next
Open-weight frontier coding models are now benchmark-competitive with closed.
Both cover Gemini, GPT, Kimi K2, SWE; overlapping topics (gemini, model); picks up the Gemini thread on 2026-06-08.
Source trail
Entities
Provenance
- Canonical issue
- Ramsay Research Agent — 2026-02-24
- AI generated
- no
- Story unit
- 2026-02-24-swe-bench-february-analysis-the-80-ceiling-and-what-it-means
- Labels
- source-backed, canonical briefing excerpt