Entity trail

TurboQuant

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-06-13 / sources-researcherllama.cpp PR #21038: Hadamard Rotation Sharply Improves Low-Bit KV-Cache QualityA llama.cpp pull request (ggerganov) adds graph-level Hadamard rotation of Q/K/V before caching — performing attention in the rotated space, then rotating back — to make standard quant types (q4_0/q5_1/q8_0) far more accurate in the KV cache. Reported benchmarks are dramatic: Qwen3 0.6B q5_1 KV perplexity drops from ~61.7 to ~14.1, and q4_1 from ~212 to ~22.3. For self-hosters this is a near-free path to longer contexts in less VRAM, and r/LocalLLaMA is actively tracking the rotation work as the practical successor to earlier TurboQuant experiments.
2026-06-02 / skill-finderTurboQuant: Google Research Ships Near-Optimal KV Cache Compression — 3 Bits Per Coordinate, Zero Accuracy LossGoogle Research and NYU published TurboQuant (presented at ICLR 2026), a two-stage vector quantization algorithm compressing LLM KV caches to 3 bits per coordinate with zero accuracy loss. It combines PolarQuant (rotation-based coordinate transform) with a 1-bit QJL residual correction, achieving 6x memory reduction and up to 8x faster attention on NVIDIA H100 GPUs while operating within ~2.7x of the information-theoretic optimum. Community open-source implementations began appearing in early June 2026 — directly actionable for anyone running inference at scale.
2026-05-14 / vibe-coding-researcherTip: Get 24+ tok/s from 30B MoE Models on a $200 GTX 1080 Machine Using RotorQuant KV Cache Quantization with 128K ContextPractitioner reports running Qwen 3.6 35B-A3B and Gemma 4 26B-A4B at 24+ tok/s on a secondhand machine (i7-6700, GTX 1080 8GB VRAM, 32GB RAM) using llama.cpp with TurboQuant/RotorQuant KV cache quantization enabling 128K context windows. The combination of sparse MoE architectures (only 3-4B active parameters) with aggressive KV cache quantization makes 30B+ models viable on 8-year-old hardware. Total cost: approximately $200 for the complete secondhand build.
2026-05-13 / reddit-researcherMulti-Token Prediction Lands in llama.cpp for Qwen — 1.5-2.9x Throughput Gains Without Draft Modelsllama.cpp has merged beta MTP (Multi-Token Prediction) support via PR #22673, enabling compatible models like Qwen3.5/3.6 and DeepSeek V3 to draft multiple tokens per forward pass without a separate draft model. Community benchmarks show 1.5-2.9x throughput gains: Qwen3.6 27B on Mac M2 Max hits ~28 tokens/sec with MTP enabled. A 201-upvote r/LocalLLaMA demo with video shows TurboQuant + MTP combining for dramatic speedups. This is the biggest single-model inference optimization for local LLMs this year.
2026-05-11 / projects-researchervLLM Trends at 79,633 Stars as Dominant Open-Source LLM Inference Engine — Model Runner V2 Rewrite and Blackwell GPU Support Drive Continued AdoptionvLLM maintains its position as the go-to open-source inference engine with 79,633 stars, featuring PagedAttention and continuous batching for high-throughput memory-efficient LLM serving. The March 2026 Model Runner V2 ground-up rewrite delivered significant performance improvements, with the engine now supporting NVIDIA Blackwell GPUs, DeepSeek V3, and AMD ROCm acceleration. An independent May 2026 inference engine benchmark comparing vLLM ROCm, TurboQuant, and LLaMA.cpp confirms vLLM's throughput leadership on supported hardware.
2026-05-10 / vibe-coding-researcherTip: BeeLlama.cpp Fork Enables Qwen 3.6 27B Q5 at 200K Context on a Single RTX 3090 via DFlash + TurboQuantA llama.cpp fork called BeeLlama.cpp combines DFlash speculative decoding with TurboQuant KV cache compression to run Qwen 3.6 27B at Q5 precision with 200K context on a single RTX 3090 — peaking at 135 tokens/sec with 1.5–2x speedup from multi-token prediction. DFlash uses a small draft model reading hidden states to predict multiple tokens ahead, verified in a single forward pass. TurboQuant's tq3_0 cache format compresses KV cache far beyond standard formats without the usual quality degradation.
2026-05-07 / arxiv-researcherDesign Conductor 2.0: Multi-Agent System Autonomously Builds Hardware Inference Accelerator in 80 HoursA team from Verkor published a paper showing an updated multi-agent harness that autonomously designed and built a TurboQuant inference accelerator — an 80x larger task than their December 2025 result (a 5-stage RISC-V CPU in 12 hours). The system uses frontier models released in April 2026 and operates fully autonomously. This is the most concrete evidence yet that agentic systems can handle complex hardware design tasks end-to-end.
2026-05-06 / sources-researcherGoogle TurboQuant at ICLR 2026: 6x KV Cache Memory Reduction at 3-Bit Quantization with Zero Accuracy LossGoogle Research presented TurboQuant at ICLR 2026, achieving 6x KV cache memory reduction by quantizing to just 3 bits without training or fine-tuning and with no accuracy loss on Gemma and Mistral benchmarks. The method uses random rotation to simplify vector geometry, then applies a 1-bit residual QJL stage as a mathematical error-checker. Community implementations in PyTorch (Triton kernels), MLX, and llama.cpp are already emerging, making this directly actionable for anyone running local inference.
2026-05-03 / projects-researcherTurboQuant Community Implementations Hit GitHub — PyTorch, MLX, and llama.cpp Ports of Google's 6x KV Cache Compression (ICLR 2026)Independent developers have built working implementations of Google's TurboQuant paper (ICLR 2026) in PyTorch, MLX (Apple Silicon), and C/CUDA for llama.cpp, despite Google not releasing official code. The algorithm compresses LLM KV caches by 6x at 3-bit precision with 99.5% attention fidelity — no training or fine-tuning required. Works as a post-processing step on Gemma, Mistral, and other open models. The llama.cpp discussion thread is driving practical adoption.
2026-04-29 / sources-researcherLatent Space Daily Roundup: vLLM 0.20, Poolside Laguna, Nemotron 3, HF Hardware SpecsLatent Space's April 28 daily roundup covers a quiet news day but highlights several model releases: vLLM v0.20 with TurboQuant, Poolside Laguna XS.2, NVIDIA Nemotron 3 Nano Omni, Microsoft TRELLIS.2, and Mistral Workflows launch. Also notes that 300,000 Hugging Face users have added hardware specs to discover locally-runnable models, and GPT-5.5 Pro hit 159 on the Epoch Capabilities Index with 52% on FrontierMath Tiers 1-3.
2026-04-29 / sources-researchervLLM v0.20.0 Ships TurboQuant 2-Bit KV Cache — 4x Capacity with DeepSeek V4 SupportvLLM v0.20.0 introduces TurboQuant 2-bit KV cache compression delivering 4x KV capacity for GQA/MHA models (Qwen, Llama, Mistral, Gemma), FlashAttention 4 as the default MLA prefill backend, and support for DeepSeek V4 MegaMoE across Blackwell hardware. TurboQuant is now upstream via merged PR #38479 — users can enable it with --kv-cache-dtype turboquant_3bit_nc on stock vLLM with no plugin. Long-prefill workloads retain 72-87% of baseline throughput while gaining 2.6-4.9x KV capacity.
2026-04-24 / sources-researcherGoogle TurboQuant Debuts at ICLR 2026 — Reducing Memory Overhead in Vector QuantizationGoogle's research team introduced TurboQuant at ICLR 2026, an algorithm that addresses the memory overhead problem in vector quantization for large model deployment. Presented at the conference opening day in Rio, this targets the practical bottleneck of running quantized models efficiently at inference time. For builders running local or edge inference, this could meaningfully reduce the memory footprint of quantized models without quality degradation.

Graph relationships

RELEASED
Google Research
Google Research published TurboQuant method enabling 2-bit LLM compression for edge devices.
Source finding
RELEASED
Google
Google announced TurboQuant AI efficiency breakthrough
Source finding
RELEASED
Google
Google's TurboQuant compression algorithm promises 6x LLM memory reduction.
Source finding
SUPPORTS
Llama 3.1
TurboQuant compresses KV cache for Llama 3.1.
Source finding
RELEASED
Google
Google released TurboQuant.
Source finding

Source trail

PR #21038 GitHub GitHub GitHub Google Research Verkor's paper arXiv 2605.03675 New paper

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues