Diffusion models decode up to 72x faster, training-free

A new decoding method cuts diffusion language model inference time by up to 72.81x on long sequences, with no extra training. Diffusion models decode several tokens per step, but each step still attends to every unfinished suffix token, a cost that grows with sequence length.

Existing speedups keep attention to a local window of suffix tokens. But they re-initialize every suffix token from scratch each timestep, discarding whatever the model already inferred, per the structured suffix modeling paper.

The new method splits the suffix into three zones: local, middle, and tail. Each zone keeps a different token count based on its role in decoding. Instead of wiping suffix representations at every step, the method carries forward the prior step's decoding results, so denoising information accumulates over time.

The method is orthogonal to parallel decoding and KV caching, the two speed tricks diffusion LMs already use. It stacks with both rather than replacing them. The paper tested the method on three diffusion language models at once.

The paper doesn't say how the speedup scales past the sequence lengths it tested. It also doesn't say whether the gains hold on models larger than the three used here.