Puro-2B pretrains a 2B model on RTX 5090s for under $6,900

Puro-2B trains a 2B parameter language model from scratch on consumer RTX 5090 GPUs for under $6,900, per the Puro-2B paper on arXiv. It runs pretraining in FP8 across up to 1.4 trillion tokens and gets close to Qwen2.5-1.5B under the authors' own evaluation protocol.

The authors cite more than $1.5 million to train Llama-3.2-3B and more than $700,000 to reproduce SmolLM3-3B. Puro-2B's budget is more than 200 times below the Llama figure, on hardware a hobbyist can own instead of renting from a cloud provider.

No single trick explains the gap. The savings stack four techniques: RTX 5090s over data-center accelerators, FP8 precision, hyperball optimization, and curriculum model averaging. Cut any one and the budget likely creeps back up, though the paper doesn't say by how much each piece contributes on its own.

What's missing is downstream testing. Coming close to Qwen2.5-1.5B on the authors' chosen benchmarks isn't the same as matching it on benchmarks a skeptical reader would pick. The paper doesn't say how the model behaves on tasks outside that evaluation set.