Tucker Attention: A Generalization of Approximate Attention Mechanisms Reduces Memory Footprint

arXiv·medium signal

Proposes Tucker Attention as a unifying framework that generalizes multiple existing approximate attention mechanisms (linear attention, low-rank attention, etc.) under a single Tucker decomposition formulation. Provides a principled way to navigate the tradeoff between memory efficiency and attention quality. Useful for practitioners who need to deploy large models on memory-constrained hardware and want a theoretically grounded basis for choosing attention approximations.