Sigmoid attention fixes a KV cache eviction gap

Learned KV cache eviction has a mismatch between training and inference. Training uses soft, differentiable gates that fade a token's contribution toward zero. Inference only saves memory once a token gets physically removed from the cache. A paper testing KV cache eviction checked whether gates trained the soft way survive the hard version.

The gap matters for anyone running long-context inference on a memory budget. A gate that behaves well during training can still fail once the cache drops that token for real.

The team ran a 2x2x2 comparison across attention type, learned gating, and positional encoding. Sigmoid attention scored worse than softmax as a plain dense language model on its own. Paired with learned gating, though, sigmoid-gated models deleted KV entries with almost no change in perplexity against their own no-eviction reference. Under a matched live-cache protocol, only the sigmoid-gated models beat the paper's own H2O and KeyDiff eviction baselines. Softmax-gated models, trained with the same gating setup, didn't.

It's a narrow finding. The paper doesn't test past GPT-2 scale, so it doesn't say whether the pattern holds for larger models running production caches, or whether softmax-gated models could close the gap with more tuning.

If it does hold at scale, teams building eviction into serving stacks can't treat gate training as a stand-in for delete-safety testing. The failure only shows up once you check the live-cache protocol, not the training loss.