SAS Trains the Sparse-Attention Selector Through the Language Modeling Loss Instead of Distilling Dense Attention
Post-training attention sparsifiers usually score context units with a lightweight selector and then apply hard top-k, which blocks gradients, so they fall back on distilling layer-wise dense attention distributions that rank by dense weight rather than by effect on predictions under a fixed budget. SAS injects the selector's continuous scores into attention logits in log form inside the softmax, using normalized softmax gates to calibrate historical context against the always-retained current block, so ordinary backpropagation updates the selector. A memory-efficient Triton kernel folds it into FlashAttention-style computation, and it beats trainable sparse baselines across reasoning, long-context, and agentic tasks with the largest gains at tight budgets.
↳ Follow the thread