Skills
NVIDIA ICMS: Petabyte-Scale KV Cache Offload Removes Long-Context Inference Bottleneck
NVIDIA's Inference Context Memory Storage (ICMS), part of the Vera Rubin platform, introduces a BlueField-4-powered storage tier with RDMA-accelerated flash between GPU HBM and general storage, sized for petabytes of shared KV cache per GPU pod. It delivers up to 5x more tokens per second and 5x better power efficiency versus traditional storage by pre-staging context back into GPU memory to eliminate decode stalls. For teams building long-running multi-agent systems with million-token context windows, ICMS removes the infrastructure bottleneck that previously forced context truncation.
↳ Follow the thread