Entity trail

GPQA Diamond

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-09-12 / skill-finderSame model weights swing 20 points across OpenRouter providers, and pinning a fallback chain caused an outageA practitioner writeup measures DeepSeek V4 Flash 0731 at 90.2% down to 75.3% on GPQA Diamond and 81.3% down to 58.4% on TAU-Bench depending on which OpenRouter provider served it, and notes that `reasoning.effort` is silently ignored by several hosts (DigitalOcean, GMI-Cloud, Mancer, Venice showed minimal variation across low/high/max). Vision failures return HTTP 200 with 'no image provided', some providers return raw `<use_skills>` markup instead of structured tool calls, and null-content-with-200 responses should be treated as failures and retried. Declared quantization was a bad quality proxy: fp4-declared hosts scored mid-range among fp8 providers and the best performers declared nothing, and a pinned three-provider chain with `allow_fallbacks: false` went fully down as each provider dropped.
2026-09-10 / github-pulse-researcherEdge0 runs a 35B MoE on Apple Silicon in 2.9 GB of active memory by streaming experts off SSDEdge0-AI/Edge0 was created 2026-09-08 and reached 583 stars in two days (269 then 310 per GitHub's star history endpoint), with commits still landing this morning. It packages SSD expert offload, Recover-LoRA, and prerouter routing prediction into an MLX-backed framework: `edge0-35b` is a 4-bit 40-layer 256-expert model built on Qwen3.5-MoE 35B-A3B needing ~2.9 GB peak active memory against a ~23 GB on-disk checkpoint, scoring 79.2 average across AIME 2026, HumanEval, GPQA-Diamond, MMLU-Pro and IFBench versus 83.2 for the fp16 base. A CUDA backend is a reserved directory, not yet shipped.
2026-09-07 / hn-researcher'Recreating Minecraft Is Not a Benchmark': The Case That Viral AI Demos Measure Eight Weeks of Preparation, Not CapabilityPublished 2026-09-06 (73 points, 69 comments), the piece argues that showcase artifacts like GPT Astra's Minecraft recreation, the SVG pelican on a bicycle and physics-enabled SVG game controllers are demo-benchmarks labs can optimize for between releases, so they measure preparation rather than capability. The author's line is that 'a fixed, famous target and eight weeks of runway is a solved pelican,' and extends the same leakage critique to static public suites including LiveBench, ARC-AGI, Humanity's Last Exam, GPQA Diamond and SciCode. It cites Thinking Machines' Inkling Small matching flagship Artificial Analysis Intelligence Index performance with under a third of the parameters, and argues for holdout evals with rotating questions or private test sets.
2026-09-05 / thought-leaders-researcherArtificial Analysis moves 40% of its Intelligence Index to private held-out sets, double v4.1The v4.2 update published September 4 adds AA-Briefcase, a private eval of agentic knowledge work on expert-built projects, and GDP.pdf, a document-reasoning test over 4,592 pages of tables, charts and footnotes. GPQA Diamond was dropped as saturated. Claude Fable 5.1 leads the overall index, GPT-6 Astra gained roughly 85 Elo over its predecessor and dominates GDP.pdf at 33.2%, and Meta ranks third among labs. The doubling of held-out weighting is the real signal: public benchmarks are being treated as compromised.
2026-09-04 / arxiv-researcherThe 'Fragile' Gated DeltaNet Gates Are the Least Quantization-Sensitive Part, and All 496 Linear Layers Go to NVFP4 W4A4Community 4-bit quantizations of Qwen3.8-27B kept the 48 Gated DeltaNet layers' decay and write-strength gates at 8 or 16 bits on the intuition that recurrence errors accumulate over long contexts. Minima tests that intuition by pushing NVFP4 W4A4 through all 496 linear layers including GDN, and matches BF16 within seed noise across 4K/32K perplexity, MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench and RULER to 64K, with a 5-task average of -0.52, the smallest footprint at 17.5 GiB, and 14-19% faster prefill. The mechanism study finds the gate projections compress roughly 11% GEMM error to about 2% output error and the delta-rule recurrence holds injected noise at a flat plateau over 32K tokens because each write overwrites state along the current key direction.
2026-08-29 / reddit-researcherNew Qwen3.8-27B GGUFs Claim BF16-Level AIME25 at 2.75 bpw by Learning the Quantizer and the Search TogetherA release post on r/LocalLLaMA (98 upvotes) publishes Qwen3.8-27B GGUFs built with two methods: GSQ (Gumbel-Softmax Quantization), post-training scalar quantization that jointly learns grid assignments and scales at 2-3 bits, and RCO (Riemannian Constrained Optimization), which assigns a quant type per tensor under a strict size budget by gradient descent on the task loss. The 3.00 bpw build (10.1 GB) matches the BF16 base on AIME25 at 100.00, lands within about a point on GPQA-Diamond (88.89 vs 89.90) and LiveCodeBench v6 (84.57 vs 85.71), and the 2.75 bpw build (9.3 GB) still hits AIME25 100.00 with a zero-shot average exceeding BF16. All three files run unmodified in llama.cpp, Ollama and LM Studio.
2026-08-27 / reddit-researcherQwen3.8-27B Quantization Benchmark: Q4_K_M Is Free, Q2 Costs 5 Points on Agentic Coding, 1-Bit Collapses to Random ChanceA Quesma benchmark posted 2026-08-26 ran Qwen3.8-27B across GPQA Diamond, IFBench, and Terminal-Bench 2.1 (89 agentic coding tasks) on L40S, H100 and H200 via Modal. Q4_K_M at 17 GB matched BF16 at 55 GB within a point on all three. UD-Q2_K_XL at 10.7 GB held instruction-following but dropped Terminal-Bench from ~77% to ~72%. Both 1-bit quants (6.2 GB) fell to 15-20% on GPQA Diamond, which is around random chance, and degraded further on longer reasoning.
2026-08-26 / projects-researcherOrnith-1.5 Ships 9B Dense, 35B MoE and 397B MoE Under MIT, With the 397B Claiming 86.0 SWE-bench Verified Against Claude Opus 4.8's 85.8DeepReinforce released the Ornith-1.5 family on August 19 under MIT in three sizes, and it is now the highest-trending non-Qwen text model on Hugging Face at a trending score of 424 for the 35B. The 397B flagship reports 86.1 Terminal-Bench 2.1, 86.0 SWE-bench Verified, 65.1 SWE-Bench Pro and 79.6 SWE-Bench Multilingual; the 35B activates only 3B parameters per token and reports 79% SWE-bench Verified and 89.2 GPQA Diamond. The training pitch is a closed self-scaffolding loop where the model proposes its own tasks, writes the scaffolds and generates its own RL rollouts, which is the part worth skepticism until someone reproduces it.
2026-08-21 / rss-researcherOrnith-1.5 ships 397B, 35B and 9B open weights under MIT and lands within a point of Claude Opus 4.8 on Terminal-BenchOrnith released Ornith-1.5 on 19 August 2026 as three MIT-licensed models: a 397B MoE, a 35B MoE, and a 9B dense with a quantized Mobile build that runs on iPhone and Android. The 397B scores 86.1 on Terminal-Bench 2.1 (Terminus-2) against Claude Opus 4.8's 85.0, 86.0 on SWE-bench Verified, and 92.8 on GPQA Diamond, but trails badly on the harder agentic rows (13.5 vs 21.1 on Frontier-Bench v0.1, 59.5 vs 69.7 on NL2Repo). All numbers are vendor-reported and averaged over five runs; weights for every size are already downloadable, unlike GLM-5.3 which is still API-only.
2026-08-20 / reddit-researcherOrnith-1.5 Ships 9B, 35B-A3B and 397B Open Weights Claiming Claude Opus 4.8 Parity, With 86.1 on Terminal-Bench 2.1The Ornith team published Ornith-1.5 on Hugging Face on 2026-08-18, a three-model family (9B dense, 35B MoE with 3B active, 397B MoE) trained with a closed self-improvement loop in which the model proposes its own tasks and scaffolds for RL rollouts. The 397B claims 86.1 on Terminal-Bench 2.1, 86 on SWE-Bench Verified, 56 on DeepSWE, 92.8 on GPQA Diamond and 44.6 on HLE, which the team frames as comparable to Claude Opus 4.8. A top commenter's side-by-side table shows the 35B-A3B losing to Qwen3.8-27B on Terminal-Bench (68.5 vs 73.0) and DeepSWE (22.0 vs 42.2) while winning NL2Repo (46.2 vs 42.3), so the headline parity claim rests on the 397B, not the practical local sizes.
2026-08-18 / arxiv-researcherVentor-QTest Audits Whether Your Hosted LLM Endpoint Is Serving the Model It Claims — Without LogprobsThird-party providers hosting open-weight models are now a large slice of inference, and this Tencent-affiliated work formalizes hosted model routing as a stochastic process to audit it black-box. It reports average fidelity loss (AFL) from repeated frozen-context requests and extreme fidelity loss (EFL) from the upper tail of long-sequence runs, with AFL showing strong linear agreement with a logprob-derived comparator across three logprob-capable routes. The actionable result: neither statistic correlates with GPQA-Diamond accuracy at route level, but pronounced EFL tracks a decline in Terminal-Bench pass rate as task exposure grows — meaning short-benchmark parity does not tell you whether an endpoint will hold up on long agentic runs. Implementation is open-source in Tencent's AI-Infra-Guard.
2026-08-16 / rss-researcherQwen ships Qwen3.8-27B under Apache 2.0 — a 27.8B dense multimodal model claiming 61.7% on SWE-Bench ProAlibaba's Tongyi Lab released Qwen3.8-27B on 2026-08-14: 27.78B dense parameters, text/image/video input, Apache 2.0, and a native 262,144-token context extensible to 1M via YaRN. Reported scores include SWE-Bench Pro 61.7%, LiveCodeBench 90.3%, GPQA Diamond 89.2%, and Terminal-Bench 2.1 at 73.0 (up from 63.4 in Qwen3.6-27B), with DeepSWE 1.1 jumping 13.3 → 42.2. It arrived 11 days after Qwen3.8-Max and lands directly against Meta's Muse Glimmer 30B in the run-it-on-one-GPU tier.

Graph relationships

DEPRECATES
Intelligence Index
GPQA Diamond was dropped from Intelligence Index v4.2 as saturated.
Source finding
BENCHMARKED AGAINST
deepseek-v4-flash
DeepSeek V4 Flash scored 90.2% to 75.3% on GPQA Diamond across providers.
Source finding
DEPRECATES
Intelligence Index -> GPQA Diamond
GPQA Diamond was dropped from Intelligence Index v4.2 as saturated.
Source finding
BENCHMARKED AGAINST
deepseek-v4-flash -> GPQA Diamond
DeepSeek V4 Flash scored 90.2% to 75.3% on GPQA Diamond across providers.
Source finding

Source trail

mmoustafa.com Willison Artificial Analysis Quesma Hugging Face Ornith AI Release Tracker TechNode reported it first Qwen3.8-Max at WAIC Shanghai GLM-5.2 Simon Willison surfaced an unsealed October 2022 email MarkTechPost

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…