Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-10 · high
The optimal key transform turns out non-orthogonal, and 2-bit quantization recovers accuracy scalar methods lose.
Why now: NOVA-KV posted to arXiv in August 2026, right as long-context agent workloads make cache bandwidth a binding cost on GPU serving.
NOVA-KV reframes key-value cache quantization as transform coding, scoring error against the attention output instead of the raw cached values, per arXiv 2608.04074. That distinction matters. Long-context decode is bandwidth-bound, and cache size sets both token throughput and how many agent sessions fit on a single GPU.
Standard scalar quantization shrinks the cache but degrades long-context retrieval accuracy. NOVA-KV instead derives closed-form optimal transforms straight from calibration statistics, built to satisfy a generalized Parseval relation, per the paper. The resulting key transform is non-orthogonal, a break from how most compression schemes are structured. At two bits per element, it recovers most of the retrieval accuracy that scalar quantization loses.
Here's the bet: if the non-orthogonal result holds outside this paper's own benchmarks, KV cache quantization stops being a storage trick. It becomes the lever that decides how many concurrent agent sessions a GPU can serve. Whoever gets transform-coded caching into a production serving stack first should pack more sessions per GPU than teams still running scalar quantization.
Each link below shares sources, entities, or timing with this story.
KittenML/KittenTTS (10.4K stars) — Sub-25MB, 15M parameter text-to-speech running on CPU, including Raspberry Pi. Commodity voice synthesis for agents without GPU requirements. When your agent needs to speak, this eliminates the API dependency. SqueezeAILab/CDLM — 14.5x infere...
arXiv 2608.08097 exploits decode-time attention sparsity: keep a 2,048-token budget resident and speculatively prefetch the rest via lookahead prediction. Reported results are 1.69x speedup on reasoning workloads, up to 2.1x on multi-GPU long-context serving, roughly 2x throug...
DCP splits KV cache across GPUs by token position rather than attention head, eliminating the duplication that caps tensor parallelism on long contexts (vLLM). On 8×B200 with Kimi K2.6, baseline TP plateaued at ~1,863 tok/s/GPU at concurrency 64 while DCP reached 6,091 tok/s/G...
arXiv 2607.27090 targets the hidden tax in Mem0, MemGPT, and Zep: retrieved memory facts get re-prefilled on every single request. It precomputes each fact's KV representation, stores it on-GPU with a semantic embedding, and injects it into vLLM's paged cache via the KV-connec...
It compresses fine-grained local features bottom-up into a small set of proxy states per layer, runs global attention only in that compressed space, then decompresses and injects back into a local stream that persists across layers, so information one compression step misses s...
arXiv 2607.29678, from Zhenyu Zhang and Zhichao Cao, profiled 153,951 real calls across two agent ecosystems and found that *even with prompt KV caching*, frontend re-tokenization of the full transcript dominates TTFT at high cache hit rates. Nobody instruments this. TokTier d...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.