Shannon Scaling Law: Information-Theoretic Framework Explains Non-Monotonic LLM Scaling Failures

arXiv·medium signal

Existing power-law scaling laws fail to explain catastrophic overtraining and quantization-induced degradation where performance worsens despite increased compute. The Shannon Scaling Law models LLM training as information transmission over a noisy channel via the Shannon-Hartley theorem, mapping model parameters to channel bandwidth. Provides a unified theoretical framework that predicts when scaling will break down — directly relevant for teams making training compute allocation decisions.