llama.cpp now runs a 295B model on one machine

llama.cpp merged native support for Tencent's Hunyuan Hy3 architecture on July 14, per PR #25395. Hy3 is a 295B-parameter mixture-of-experts model with 21B active parameters. It now loads on a single high-memory box instead of a multi-GPU cluster, cutting the entry ticket for teams running private or high-volume inference.

Community GGUF quants from AngelSlim and others are already on Hugging Face. One 1M-context conversion reportedly runs at about 11 tokens per second on a single box. The PR also wired up MTP speculative decoding, measured at roughly 40% faster decode throughput on code. That's the kind of detail that matters more than the headline number for anyone actually running this thing.

This isn't a one-off. It's a July pattern. Tencent shipped 1-bit and 4-bit Hunyuan 3 builds meant for a single GPU. PrismML released Bonsai 27B as a 1-bit build at 3.9GB, claiming it keeps multimodal and agentic capability under Apache 2.0, per Latent Space. NVIDIA leaned into the same direction with Nemotron Labs, positioning open models as something enterprises and nations can run and control on-prem, per NVIDIA.

The old math said frontier-scale meant a multi-GPU cluster and a five-figure entry ticket. That's no longer true for a growing class of models. The catch is real: 1-bit and Q2 quants lose quality. And 11 tok/s at 1M context works for overnight batch jobs, not for anyone waiting on a reply.

What I'd do

If you're paying per-token for work that doesn't need frontier quality, price a single high-memory rig against your monthly API bill. The cloud doesn't win by default anymore. Test a Q4_K_M Hy3 or Bonsai 27B build against your actual workload. For regulated or sensitive data, keeping the model on your own box means it never phones your data to a vendor.