Reduced Matrix Multiplication cuts Transformer inference with no weight changes, and attention is far more reducible than MLPs.

RMM is training-free and input-adaptive, selecting informative slices along contraction dimensions under a single retention-ratio knob. Tested from 1B to 70B across discriminative, autoregressive and long-context settings, reduction tolerance often improved with scale. Custom A100 kernels turned theoretical savings into wall-clock gains, especially at long sequences. arXiv 2608.13426