DnA: Denoising Attention for Visual Tasks

arXiv·low signalXBlueskyLinkedInCopy link

Replaces the de facto softmax activation in multi-head attention with a denoising attention mechanism to reduce the noisy attention patterns that softmax can produce in visual perception tasks. The drop-in change targets cleaner attention maps and improved perception accuracy. A lightweight technique for vision-model builders.