OSS
mesh-llm v0.76.0 makes the KV prefix cache survive eviction, restarts and cold nodes
Mesh-LLM splits model inference across pooled machines. Its v0.76.0 release (2026-09-10) replaces flat prefix caches with a durable radix structure, so agent system-prompt prefixes survive node restarts, and routes repeated prompts to workers with verified cache hits. The release also streams tool calls mid-generation, adds staged Qwen3.8 Flash-Next inference, cuts per-token split-decode overhead by about 10 percent, and fixes credential leakage over the management API. The repo has 3,375 stars, with 50 open PRs against 100 open issues.
Source
↳ Follow the thread