vLLM ships multi-tier KV-cache offloading and batch-invariant FP8 for a 28.9% latency cut

vLLM·medium signal

vLLM's June 2026 releases add a multi-tier KV-cache offloading framework (Python filesystem secondary tier plus Mooncake disk offload, extending beyond CPU memory), make Model Runner V2 the default for Qwen3 dense models, and land Cutlass FP8 for batch-invariant inference yielding a 28.9% end-to-end latency improvement. For anyone self-hosting LLM inference, the KV offloading tiers let you serve longer contexts and higher concurrency on the same GPUs. Batch-invariant FP8 also gives you reproducible outputs across batch sizes — useful for evals and caching.