Shannon Scaling Law Explains Why Scaling Sometimes Makes Models Worse.

This paper models LLM training as information transmission over a noisy channel via the Shannon-Hartley theorem. Existing power-law scaling laws can't explain catastrophic overtraining or quantization-induced degradation. This framework predicts when scaling breaks down. Directly useful for teams making training compute allocation decisions.