Make KV cache elastic with kvcached to share one GPU across multiple LLMs (2–28× faster TTFT)

GitHub·high signal

kvcached decouples GPU virtual addressing from physical memory so serving engines reserve virtual KV-cache space at startup but only back it with physical memory when actually used, enabling several models to share one GPU instead of statically pre-reserving it. Turn it on as a transparent plugin — no code changes — by setting ENABLE_KVCACHED=true and KVCACHED_AUTOPATCH=1 on vLLM (v0.8.4+) or SGLang (v0.4.9+); it also adds cross-request prefix reuse via APC (vLLM) or RadixCache (SGLang). Berkeley's tests with three Llama-3.1-8B models on one A100-80G show 2–28× TTFT reduction under bursty workloads.