llama.cpp's RPC server was caching activations as if they were weights, filling a disk with 14,802 files and 1.4 TB
PR #28789, merged 2026-09-15, found that ggml_backend_rpc_buffer_set_tensor hashed every transfer above the 10 MB threshold and let rpc-server -c serve it from the file cache, which was intended for weights but also caught the activations ggml_backend_sched copies between backends. With a two-node split of Qwen3.8-Flash-Next, every prefill ubatch above 10 MB was hashed and written to the worker's cache; after a day the cache held 14,802 files and 1.4 TB. Once the disk filled, the server wrote truncated cache files and later served them as complete, producing Inf/NaN activations, which is how the bug was found. The fix gates hashing on GGML_BACKEND_BUFFER_USAGE_WEIGHTS, adds a cache_flag byte to SET_TENSOR and bumps RPC_PROTO_MAJOR_VERSION to 7; a 16 MiB compute tensor drops from 19 ms to 2.9 ms.
Source
↳ Follow the thread