MISA: Drop-in Replacement for DeepSeek Sparse Attention Cuts Long-Context Indexer Cost via MoE Routing

arXiv·medium signal

MISA treats DeepSeek Sparse Attention's 64 indexer heads as a mixture-of-experts pool with a lightweight router that selects only a few active heads per query using block-level statistics. This preserves indexer diversity while reducing the per-query cost from scoring every prefix token with every head to scoring with only routed heads — making long-context inference significantly cheaper.