Longer training context makes models worse at recall

A new paper posted to arXiv, 2608.12218, finds that pretraining gains from longer context reverse once length passes an intermediate optimum. Push training documents past that point and language modeling, natural language understanding, and closed-book multiple-choice QA all get worse, not better.

That closed-book recall is what a model falls back on when retrieval fails or a prompt arrives thin. If long-context pretraining quietly trades that fallback for context-following, it's a design choice buried inside a hyperparameter most teams treat as strictly more-is-better.

Researchers trace the effect to gradient pressure. Informative context pulls training weight away from feed-forward networks, the layers tied to parametric knowledge, and toward attention, which reads the current input. Causal interventions confirm the shift: models trained this way lean harder on context at inference time instead of stored knowledge.

Fine-tuning shows the same split. Extra task-relevant context at train time helps when a model actually gets supporting context at test time. It hurts robustness when that context is missing or wrong, so a model tuned to expect a well-stocked prompt gets worse at working without one.

It doesn't give the actual optimum context length, or say whether that point shifts as models scale up. That's the number anyone evaluating a pretraining run will want next.