Sliding window attention beats post-trained linear attention on retrieval

Sliding window attention with sinks beats post-trained linear attention on long-context retrieval. It scores 2 to 10 times higher on the Needle-in-a-Haystack and BABILong benchmarks, per a new comparison of attention mechanisms that tested both approaches across multiple LLMs and downstream tasks.

That gap matters for anyone weighing how to cut inference memory costs on long-context models. Linear attention has become a popular retrofit for that exact problem: take an existing model, post-train it into a linear-attention variant, and save memory without retraining from scratch.

The comparison ran the baseline retrofit papers tend to skip. It tested SWA with sinks directly against post-trained linear models on the same tasks, not just the two retrieval benchmarks but the broader set of downstream tasks too. SWA matched or beat linear attention across the board.

The paper's recommendation is blunt. Switch to SWA for the memory savings. Matching its retrieval scores with a linear model would take training from scratch or heavy post-training, the paper says, not a light retrofit.

It's a negative result for a popular idea. Papers proposing linear-attention retrofits don't tend to run this comparison, which is exactly why it's useful now.