Google TurboQuant: 6x KV Cache Compression at 3-4 Bits per Element — No Retraining, Llama 3.1 70B Drops from 40GB to 7.5GB at 128K Context

Google Research Blog·high signal

Google Research's TurboQuant (ICLR 2026) uses PolarQuant rotation plus 1-bit QJL residual correction to compress KV caches to 3-4 bits per element with near-lossless quality, achieving 4-6x memory reduction without any retraining or calibration data. Llama 3.1 70B at 128K context goes from ~40GB to ~7.5GB. On H100 GPUs, 4-bit TurboQuant delivers 8x speedup on attention logits via reduced memory bandwidth. An open-source llama.cpp implementation already exists. For builders running local models: this is the single biggest practical improvement to long-context inference memory since PagedAttention.