Super Weights Don't Generalize, and Training Them in Isolation Destroys the Model

arXiv·medium signal

Prior work identified Super Weights — individual parameters whose removal collapses LLM performance by orders of magnitude. This paper (arXiv 2607.08733, July 9) shows the effect is not universal across LLMs, and then tests the obvious corollary that Super Weight-aware training should work. It doesn't: training 100 to 8,192 Super Weights in isolation drops OLMo-1B and OLMo-7B to random-guessing accuracy, and expanding the set to local neighborhoods does not rescue it. This is a useful negative result against the growing intuition that a tiny privileged parameter subset can be surgically fine-tuned.