ovg-project/kvcached v0.1.5: OS-Style Virtual Memory for GPU KV Cache — 2-28x TTFT Reduction on A100

GitHub·medium signal

Released April 7, kvcached brings OS-style virtual memory abstraction to LLM serving, decoupling GPU virtual addressing from physical memory allocation for KV caches. Engines reserve virtual memory initially, with physical GPU memory allocated on-demand — enabling multiple models to elastically share GPU memory without rigid partitioning. Benchmarks show 2-28x TTFT (Time to First Token) reduction for three concurrent Llama-3.1-8B models on A100-80G, with support for SGLang and vLLM serving engines.