Splitting Decode by Attention Type Instead of by Operator Buys 31-56% More Tokens Per Joule

arXiv 2609.13134·medium signalXBlueskyLinkedInCopy link

SQD argues that disaggregated serving built around dense attention is the wrong shape for models using sparse, linear, or sliding-window attention, and instead splits decode by quadratic versus subquadratic work. For sparse-attention models it separates top-k selection, which must index the full KV cache, from top-k attention plus FFN, which have static memory footprints. On an adjusted 8xB200 heterogeneous proxy it measures 53% better tokens per joule on GLM 5.2, 31% on Nemotron 3 Ultra, and 56% on Gemma 4 31B, and an analytical Rubin-plus-LPX model shows up to 3.6x higher throughput over attention-FFN disaggregation.