TRL 1.13.0 trains Qwen3-8B on a 1,048,576-token sequence on one 8xH100 node and makes chunked-CE 6x faster

GitHub·high signalXBlueskyLinkedInCopy link

Hugging Face TRL v1.13.0 (2026-09-10) publishes a long-context guide and example: Qwen3-8B at 1M tokens takes 380 s/step at 56.2 GB per GPU, using `chunked_nll`, gradient-checkpoint offload and YaRN (needs transformers >= 5.16). It also removes a pointless fp32 upcast in the chunked-CE lm_head GEMM. That upcast accounted for 21.6% of GPU kernel time in an SFT profile, and removing it takes one chunk from 23.37 ms to 3.86 ms. End to end, Qwen3-8B full fine-tuning goes from 3,554 to 6,009 tokens/s/GPU (1.69x).