Entity trail

Qwen4

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-09-05 / github-pulse-researcherllama.cpp 0.4.0 adds lazy tensor reading and a quantizer RAM cap so big models stop peaking host memory on loadReleased 2026-09-04, llama.cpp 0.4.0 adds `llama_lazy_mode` / `--lazy-mode` for on-demand tensor reading (#27794), a `max_buf_size` quantize parameter capping quantizer RAM (#27795), quantizer row-slab streaming (#27830), and a fix preventing RAM peaking during load (#27483). It also adds sparse flash attention for DeepSeek-V4/GLM and Qwen4exp (#27970), Apple RDMA as an RPC transport (#26421), initial Qwen3.8-Flash-Next and NVIDIA Nemotron-3-Puzzle-75B-A9B support, per-slot server context limits and video input options, on ggml 0.23.0.
2026-09-01 / github-pulse-researcherllama.cpp replaces a transpose-plus-sum_rows indexer reduction with strided views, worth 9% prefill on an RTX PRO 6000Build b10730 (#28023), published 2026-09-01, rewrites the qwen4exp indexer head reduction that went through a transpose and a sum_rows over ne[1], leaving sum_rows with ne0 = 4 — one block per row for a four-element reduction — while the transpose copied the whole block-by-token surface twice on the way in. Because the heads are adjacent on ne[1], each is now a strided view and the sum is a short chain of adds. On an RTX PRO 6000 with Qwen3.8-Flash-Next UD-Q4_K_XL, fa on and 55k context, prompt processing goes from 2,170 to 2,366 t/s with generation unaffected, and the gain grows with context and ubatch size.
2026-09-01 / github-pulse-researcherllama.cpp adds recurrent state rollback for Qwen3.8-Flash-Next, taking MTP speculative decoding from 123 to 183 tok/sBuild b10731, published 2026-09-01, implements recurrent state rollback for qwen4exp (#28123), which MTP speculative decoding needs so the target state can move back by the number of rejected draft tokens. Without it the context was classified SEQ_RM_TYPE_FULL and the server serialized the whole recurrent state to host memory every round, costing more than the drafting saved. Measured on Qwen3.8-Flash-Next UD-Q4_K_XL with n-max 3 and a single slot, decoding reaches 183 tok/s on code and 144 on prose, against 123 and 83 on the same branch before the change and 108 tok/s with no draft at all.
2026-08-28 / rss-researcherQwen3.8-Flash-Next is a 125B/6B-active multimodal MoE and an early look at the Qwen4 architectureQwen released the open-weights model on 2026-08-26, describing it as a multimodal mixture-of-experts with 125 billion total parameters and 6 billion active, explicitly framed as a preview of the architecture Qwen4 will use. Simon Willison ran quantized Unsloth builds on a DGX Spark, a 72.5GB UD-IQ1_S and a 78.9GB UD-Q2_K_XL, both on Hugging Face, and got his best result from UD-Q2_K_XL at xhigh reasoning effort. The 6B active count is the number that matters for local inference: it puts a 125B multimodal model inside a single workstation's reach.
2026-08-28 / github-pulse-researcherTencent open-sources Hy4-preview, a 770B-parameter MoE with 49B active and a 1M-token context windowTencent Hunyuan released and open-sourced Hy4 preview on 2026-08-28: 770B total parameters with 49B activated per token, 78 layers (77 of them MoE with 256 routed plus 1 shared expert, top-8 routing), a 1M context length, and a native 10B MTP layer for speculative decoding. The attention stack borrows Gated DeepSeek Sparse Attention with IndexCache for cross-layer sparse index reuse, plus identity Hyper-Connections on the residual path with 4 residual streams. Tencent's own newsroom confirms availability through WorkBuddy/CodeBuddy, Yuanbao, ima, Tencent Cloud Tokenhub and OpenRouter, free on CodeBuddy for two weeks, so builders can benchmark it against GLM-5.3 and Qwen4 without hosting 770B weights.
2026-08-27 / github-pulse-researchertransformers v5.16.0 adds Qwen4-Exp, the first hybrid architecture combining linear and sparse attentionReleased 2026-08-26, v5.16.0 adds Qwen4-Exp with three new components: GatedResidual, which mixes multiple residual streams with elementwise gating before each attention and MoE block; Qwen Sparse Attention, which scores compressed key blocks and selects contiguous token blocks while leaving the trailing block uncompressed; and Per-Layer Embedding, which enriches selected decoder layers with hashed token n-gram features. Combined with Gated DeltaNet, QSA makes Qwen4-Exp the first hybrid architecture to integrate both linear and sparse attention. The block-level selection is aimed squarely at long-context inference cost, which is the binding constraint for agent workloads.
2026-08-27 / github-pulse-researcherQwen releases Qwen3.8-Flash-Next weights as an early architecture preview of Qwen4, and ollama shipped MLX support two days laterQwenLM/Qwen3.8-Flash-Next appeared on GitHub 2026-08-24 and is at 179 stars, opening the weights of a multimodal MoE that Qwen explicitly frames as an early preview of the Qwen4 architecture, the same role Qwen3-Next played for Qwen3.5. The hybrid Gated DeltaNet plus Gated Attention design it previews has already carried through the Qwen3.5 to Qwen3.8 series. The release is corroborated downstream: ollama v0.33.1 (2026-08-26) lists MLX Qwen3.8 Flash Next support, and transformers v5.16.0 added the related Qwen4-Exp with Qwen Sparse Attention and Per-Layer Embedding.
2026-08-27 / reddit-researcherr/LocalLLaMA Reverse-Engineered Qwen4's N-Gram Architecture: MoE Does Reasoning, N-Grams Do Recall, and Only RAM WorksA 217-upvote r/LocalLLaMA writeup breaks down the Qwen4Exp architecture behind Qwen3.8-Flash-Next: a router picks experts late in the layer with a large payload, so experts resist offload, while an n-gram table is a hash lookup known early and cheap to fetch. The author's conclusion is that roughly 25 percent of weights can move to n-gram before the tradeoff inverts, turning a 176B model into 125B in RAM plus 51B elsewhere. The thread's correction to the original SSD claim matters for builders: the n-gram index is hashed, so access is random and RAM is the only viable tier.
2026-08-26 / reddit-researcherQwen3.8-Flash-Next Got a Release-Day Megathread and Unsloth Day-0 Quants Before the Weights Existedr/LocalLLaMA moderators pinned a release-day megathread for Qwen3.8-Flash-Next with an estimated drop of 2026-08-26 15:00 UTC, pointing at the Hugging Face and ModelScope pages, while Unsloth's day-0 quant announcement pulled 717 upvotes with a top comment thread of people who had just finished tuning 3.8-27B ('paint is still wet on 27b'). The architecture is the headline: a Qwen4-preview multimodal MoE with GDN gated-delta hybrid layers and Qwen Sparse Attention, reported at 125B main parameters plus 51B N-gram embeddings and 6B active per token, with a claimed ~1/9 the training cost of Qwen3.7-Plus at comparable capability. Commenters are already asking for llama.cpp flags to place sparse KV cache on SSD (n-ssd-ffn, n-cpu-ffn), which is the practical bottleneck for running this locally.
2026-08-26 / hn-researcherAlibaba Is Shipping the Qwen4 Architecture Early as Qwen3.8-Flash-Next, 125B Total With 6B Active and No Benchmarks YetQwen3.8-Flash-Next, an open-weight multimodal mixture-of-experts model, is scheduled to go public on ModelScope at 23:00 Beijing time on August 26, 2026. It is roughly 125B parameters with a separate N-gram embedding table of about 51B, activating only 6B per token, and Alibaba frames it as a technology preview of the architecture that will power Qwen4. Alibaba claims training cost around one ninth of Qwen3.7-Plus, but has published no side-by-side scores against its own line or Western rivals, so treat the parameter and efficiency figures as unverified until the weights land.
2026-08-25 / reddit-researcherQwen Staged Qwen3.8-Flash-Next on ModelScope as a Qwen4 Architecture Preview, 125B Main Params Plus 51B N-Gram Embeddings and 6B ActiveThe ModelScope card that went up ahead of release described a redesigned multimodal MoE with 125B main-model parameters, an additional 51B of n-gram embeddings, and 6B active per token, stating it is built on the next-generation Qwen4 architecture and released early so the community can prepare for the Qwen4 family. Commenters on the 369-upvote r/LocalLLaMA thread captured screenshots before Qwen edited paragraphs out of the readme minutes later, and noted an FP8 version was listed with no FP4 or QAT-only release. The precedent worth planning around is Qwen3-Next, whose novel architecture took roughly two months to land in llama.cpp.

Graph relationships

BENCHMARKED AGAINST
Hy4
Hy4 can be benchmarked against Qwen4
Source finding
BENCHMARKED AGAINST
Hy4 -> Qwen4
Hy4 can be benchmarked against Qwen4
Source finding

Source trail

Hugging Face simonwillison.net GitHub GitHub r/LocalLLaMA r/LocalLLaMA Staged on ModelScope GitHub GitHub GitHub GitHub r/LocalLLaMA megathread, weights at huggingface.co/Qwen/Qwen3.8-Flash-Next

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…