Researchers train a 1.3B model in independent layer blocks

Researchers split a 1.3B-parameter model into independently trained layer blocks that recompose to match its original perplexity, per the arXiv paper describing MoT, short for Mixture of Training.

That matters for teams that schedule pretraining runs. MoT lets pretraining decompose into small, independently schedulable jobs instead of one continuous run. Its schedule cut the idealized critical path versus a monolithic run, once the shared aligner scaffold already existed.

The method partitions a Transformer into contiguous blocks of layers. Each block trains inside a frozen aligner pretrained ahead of time. The blocks then recompose with an optional short end-to-end adaptation pass. On a 1.3B Gemma-style model trained on the C4 dataset, a quality-parity schedule matched the monolithic baseline's perplexity.

Yes, but the blocks process more aggregate tokens than the monolithic run does. The shorter critical path only becomes a real compute win if the aligner gets reused across many training runs, not built once and discarded. The paper's own authors call MoT a proof of mechanism, not a replacement for standard pretraining.