llama.cpp intelligence trail

Dossier

Compiled 2026-09-09 · source-backed

github.com · 21 www.reddit.com · 13 huggingface.co · 2 blogs.nvidia.com · 1 dev.to · 1 terminalbytes.com · 1 zenodo.org · 1

Corpus findings

2026-09-09 / github-pulse-researcherllama.cpp redefined lazy tensor loading `auto` and turned it off on integrated GPUsPR #28326, merged 2026-09-08T16:05Z, changes lazy mode `auto` to mean "pick a probably good mode for your system" rather than its old behaviour of lazily loading tensors over 4 GiB, which becomes mode `large`; loading all of them becomes mode `all`. The stated reason (issue #28160) is that the old default effectively re-enabled mmap on iGPUs, which lose substantial performance that way. A day later #28334 removed --mmap, --mlock and --direct-io from the arg parser entirely, finishing the migration to --load-mode, so scripts still passing the old flags now fail to parse.
2026-09-08 / github-pulse-researcherllama.cpp's multi-model server hung on concurrent requests for the same model until a scheduler tick was addedPR #28539, merged 2026-09-07T13:50Z, fixes an LRU hang in llama-server when several requests target the same model, closing #28443. The fix adds a tick() to the scheduler to make it more coroutine-like rather than restructuring the threading. The maintainer notes in the PR that server_models is very IO-bound and hard to maintain as a thread model, and that converting it to coroutines is under consideration, so anyone running llama-server in multi-model mode should expect further churn there.
2026-09-08 / github-pulse-researcherllama.cpp's Vulkan backend fuses DeepSeek-V4 hyper-connection ops that were eating 32% of decode time on Strix HaloPR #26578, merged 2026-09-07T13:24Z, implements DSV4_HC_COMB, DSV4_HC_PRE and DSV4_HC_POST for Vulkan, the last major backend without them after CUDA and Metal. On DeepSeek-V4-Flash the unfused Sinkhorn comb chain alone was roughly 32% of decode op time on gfx1151, spread over about 16k dispatches per token; the fused shader runs the full 20-iteration Sinkhorn in registers using subgroupShuffleXor within 16-lane blocks, replacing about 137 strictly ordered node executions per site with one dispatch. Verified against a float64 reference and the official modeling_deepseek_v4.py to about 6e-7.
2026-09-08 / github-pulse-researcherllama.cpp enabled speculative-decoding state rollback for Kimi-K3, which had it switched offPR #28466, merged 2026-09-08T03:31Z, adds Kimi-K3 to llm_arch_supports_rs_rollback and saves the convolution windows for each rollback position plus KDA state snapshots via build_recurrent_attn. Kimi-K3 previously stored only the final KDA state, so enabling rollback without those writes would have restored unwritten snapshot groups after rejecting draft tokens. The author documents that with only the allowlist change the zero-filled pass passed but the nonzero pass failed split replay at 2.35e-6 against a 1e-7 tolerance, and both pass with the snapshot writes. Local speculative decoding on Kimi-K3 was quietly unavailable until this landed.
2026-09-08 / reddit-researcherThe case against Ollama topped r/LocalLLaMA with 1,075 upvotes and 328 commentsThe post argues Ollama went over a year without crediting llama.cpp in its README while a license-compliance issue sat 400+ days without a maintainer response, that llama.cpp runs 1.8x faster (161 vs 89 tokens/second) with 30-50% CPU gaps, and that the mid-2025 move to a custom GGML backend reintroduced broken structured output, vision failures and assertion crashes. It also cites CVE-2025-51471 for token exfiltration via malicious registries and the DeepSeek-R1 distill naming. Top comments converge on LM Studio, llama.cpp and Unsloth Studio, with the recurring complaint being that Ollama will not use models already on disk.
2026-09-07 / github-pulse-researcherllama.cpp lands Qwen3-Next / Qwen3.5 conversion metadata, a QKV fusion flag, and two CUDA race fixes in 36 hoursggml-org/llama.cpp cut builds b10833 through b10839 between 2026-09-07T06:49Z and 11:14Z. #28208 writes explicit `recurrent_layers` during Qwen3-Next / Qwen3.5 HF-to-GGUF conversion, and #22780 adds a `--fuse-qkv` flag to fuse Q/K/V into a single QKV tensor at conversion time. Two correctness fixes touch CUDA concurrency: #28475 fixes races in `mmid` and `mmf`, and #27870 fixes a divergent barrier in f16 flash attention. Vulkan gained TQ1_0 support (#27765), type-aligned GET_ROWS (#28253) and rms_norm fusion (#28024), and #28068 corrects GDN normalization from `max` to `rsqrt`.
2026-09-07 / reddit-researcherA hands-on Q8 comparison puts DeepSeek-V4-Flash-Vision at 40% slower token generation but 2x faster task completion than Qwen3.8-Flash-NextAn r/LocalLLaMA practitioner running 2x Strix Halo 128GB over USB-C 4 with llama.cpp RPC compared both models at Q8_K_XL on real coding work (79 upvotes, 46 comments). DS-V4-Flash-Vision generates tokens ~40% slower yet finishes tasks about twice as fast, which the poster attributes to fewer hallucinated detours: a task Qwen finished in 25 minutes on medium took DeepSeek 12 minutes. The sharpest number is a failure mode, not a win: Qwen3.8-Flash-Next on 'xhigh' failed to complete in ~3 hours a task it finished in 25 minutes on 'medium,' so raising reasoning effort on that model is actively counterproductive for agentic coding.
2026-09-06 / reddit-researcherA block KV cache streaming PR promises bounded VRAM at long context, and testers say the swapped-out context comes back as gibberishPR #357 on the llama-cpp-turboquant fork ports Raymond Huang's adaptive KV streaming work, extends it to turboX and models beyond Qwen, and shares a CUDA phase arena to bound VRAM as context grows, the idea being you set a 1M context and see normal speed until you exceed what fits in VRAM. Two testers in the thread report it does not behave as described: the swapped-to-RAM portion of the context reads back as repeated gibberish and even truncates earlier context, so the model effectively only sees what is resident on the GPU. Worth tracking, not worth deploying.
2026-09-06 / reddit-researcherA production RTX 5090 bake-off pits NInfer and vLLM NVFP4 against llama.cpp GGUF on a six-tier custom evalA builder serving Qwen3.8-27B for a content-intelligence pipeline moved off llama.cpp specifically because it is capped at parallel=1 and they needed concurrent serving, then evaluated vLLM and NInfer NVFP4 replacements on six 50-item tiers drawn from production work rather than public benchmarks: relevance classification, needle retrieval at 64K/128K/192K/240K, multi-transcript QA over 120K+ tokens with unanswerable controls, thinking-mode reasoning, structured extraction, and replayed agent episodes. The configuration detail that matters is NInfer running MTP3 at 76% acceptance in 30.5 GB and llama.cpp's MTP gate failing outright, and NInfer not supporting json_mode at all, which disqualifies it for structured extraction.
2026-09-06 / rss-researcherGLM-5.3's reasoning_effort silently defaults to max, including when you typo itA practitioner writeup measured what it actually takes to run Z.ai's GLM-5.3 (753.9B, weights out August 28) and GLM-5.3-Flash (320.8B with 18B active, out August 27) locally. Smallest usable quants are 216.7 GB for the flagship and 93.1 GB for Flash, and counterintuitively the 753.9B flagship runs in stock llama.cpp while Flash does not yet. The trap: `reasoning_effort` takes low, high and max, and the model card says it defaults to max if unset or set to any other value, so a typo buys a long think on every message. Through a hosted OpenAI-compatible provider the behavior differed, with `medium` honored as a real middle rung and an omitted field landing at 30 output tokens rather than max's 48.
2026-09-05 / github-pulse-researcherllama.cpp 0.4.0 adds lazy tensor reading and a quantizer RAM cap so big models stop peaking host memory on loadReleased 2026-09-04, llama.cpp 0.4.0 adds `llama_lazy_mode` / `--lazy-mode` for on-demand tensor reading (#27794), a `max_buf_size` quantize parameter capping quantizer RAM (#27795), quantizer row-slab streaming (#27830), and a fix preventing RAM peaking during load (#27483). It also adds sparse flash attention for DeepSeek-V4/GLM and Qwen4exp (#27970), Apple RDMA as an RPC transport (#26421), initial Qwen3.8-Flash-Next and NVIDIA Nemotron-3-Puzzle-75B-A9B support, per-slot server context limits and video input options, on ggml 0.23.0.
2026-09-05 / reddit-researcherBase-3 packing for ternary GGUFs cuts weight VRAM about 22% with no precision lossA new llama.cpp fork (based on commit 4e97ac86e) adds Q2_B3, a format that packs ternary weights directly in base 3: 128 weights per block become 26 bytes of trits plus one f16 scale, 28 bytes total, or 1.75 bits per weight. On genuinely ternary models such as BitNet-b1.58 or Ternary-Bonsai that takes a 9B from roughly 2.5GB to 2.0GB and a 27B from 7.6GB to 5.9GB, weights only. The stated caveat matters: this is not a general 2-bit quantizer, feeding it an FP16 model destroys quality, and the CUDA path compiles but is unverified since the author only owns a 7900 XTX.

Graph relationships

COMPETES WITH
Ollama
Ollama runs 1.8x slower than llama.cpp at 89 vs 161 tokens/second.
Source finding
SUPPORTS
Kimi K3
llama.cpp enables speculative-decoding state rollback for Kimi-K3
Source finding
SUPPORTS
DeepSeek V4
llama.cpp Vulkan backend fuses DeepSeek-V4 ops reducing decode time by 32%
Source finding
SUPPORTS
Qwen3.5
llama.cpp added HF-to-GGUF conversion support for Qwen3.5.
Source finding
SUPPORTS
Qwen3-Next
llama.cpp added HF-to-GGUF conversion support for Qwen3-Next.
Source finding
BENCHMARKED AGAINST
NInfer
NInfer and llama.cpp compared on concurrent serving; llama.cpp capped at parallel=1
Source finding
SUPPORTS
Q2_B3
A llama.cpp fork added support for Q2_B3 base-3 ternary weight packing format.
Source finding
SUPPORTS
Qwen3.8-Flash
llama.cpp 0.4.0 adds initial support for Qwen3.8-Flash-Next.
Source finding
USES
GGML
llama.cpp 0.4.0 runs on ggml 0.23.0 with lazy tensor reading and quantizer RAM capping.
Source finding
SUPPORTS
DeepSeek V4
llama.cpp 0.4.0 adds support for DeepSeek-V4 with sparse flash attention.
Source finding
USES
Vulkan
llama.cpp achieves 5x speedup on Vulkan IQ3_S at batch 8
Source finding
USES
Hexagon HMX
llama.cpp fuses QKV and FFN matmuls on Qualcomm Hexagon HMX
Source finding

Source trail

GitHub GitHub GitHub GitHub r/LocalLLaMA (1,075 upvotes, 328 comments)GitHub r/LocalLLaMA (79 upvotes, 46 comments)GitHub (via r/LocalLLaMA, 30 upvotes)r/LocalLLaMA (64 upvotes)DEV Community (model cards and Unsloth GGUF pack dates cited as primaries)GitHub r/LocalLLaMA

Graph sources

entity graphfindings textkg edgeskg entities