py-kvcache Loads KV Cache From NVMe 2x Faster Than LMCache at 80k Tokens, but Loses on an H100

arXiv·medium signalXBlueskyLinkedInCopy link

arXiv 2609.11744 characterizes vLLM KV offload across GPU, CPU and NVMe tiers and releases py-kvcache, a connector with async direct I/O and scheduler-aware preloading that starts disk reads while requests are still queued. At 80k tokens it loads 2.0x faster than LMCache, and preloading accounts for 1.34x of that. On Bailian production traces it helped a weaker GPU, but on an H100 the average request fell below break-even and recomputing was faster. Treat external KV caching as a per-setup admission decision. Code is on GitHub.