Distilled Byte Models Match a Token Model's Accuracy on One-Sixth the Data and Cut Logit Storage to a Fifth

arXiv 2609.12303·medium signalXBlueskyLinkedInCopy link

This sweep trains roughly 1B-parameter layer-parameter-matched decoders across tokenization scheme (tokens, bytes, bytes with end-of-token) and objective (distillation versus cross-entropy) up to 1 trillion bytes of data, using two new methods (Marginalize-It and End-Of-Token) to convert token logits into byte logits. Token models win in the low-FLOP regime then plateau, while byte models start worse and pass them with more compute, with scaling-law extrapolation predicting distilled End-Of-Token-1B beats distilled Token-1B by up to 4% asymptotically. Operating over 256 bytes instead of a ~100K vocabulary also removes the need for top-k truncation when dumping logits and cuts logit storage to roughly one-fifth.