Mesh LLM: Distributed P2P Inference on iroh Splits Large Models Across Consumer GPUs

iroh (via Hacker News)·high signal

Mesh LLM, posted July 11 (293 points on Hacker News), distributes LLM inference across a peer-to-peer network built on iroh, using QUIC transport and hole-punching to route requests directly between nodes with no central server. Large models are partitioned by layer ranges (e.g. layers 0–15 on one machine, 16–31 on the next) so several modest GPUs can run a model none could hold alone. Most compelling for home labs and small teams hitting VRAM ceilings on local models.