kvcached at 1.1K Stars With +346 Today: Virtualized Elastic KV Cache Enables 3 Llama-3.1-8B Models on a Single A100 With 2-28x TTFT Reduction

GitHub·medium signal

ovg-project/kvcached applies OS-style virtual memory abstraction to LLM systems, enabling elastic on-demand KV cache allocation instead of static GPU memory pre-allocation. The v0.1.5 release supports SGLang v0.4.9+ and vLLM v0.8.4+ with prefix caching, and recent PRs add tiered cross-GPU KV cache memory mapping via NVLink for scale-up inference. Trusted by Intel, AMD, Red Hat, ByteDance, Alibaba Cloud, and Tencent — Red Hat built Sardeenz on top for Kubernetes/OpenShift multi-model serving.