A replay of 68,266 real Claude Code requests says plain LRU beats the clever KV-cache policies

GitHub·high signalXBlueskyLinkedInCopy link

agentic-kv-cache simulates cross-request prefix caching against real traces: 68,266 requests across 393 Claude Code sessions at 64-token blocks, plus Mooncake traces at 512-token blocks. It models prefix-contiguous hits, radix eviction constraints and pinned in-flight chains, then compares LRU-leaf against TTL-300s, LFU-leaf and a hazard-prediction policy with physically-modelled recompute costs. At 40,000 blocks LRU-leaf wins, TTL-300s is byte-identical to it, and the hazard policy degrades everywhere. The reason: 33.1% of recompute tokens come from sub-10-second gaps inside tool loops and only 17.5% from gaps over five minutes, so capacity pressure dominates, not liveness prediction.