SPADE cuts cloud AI calls 76% with no accuracy loss

SPADE cuts cloud model calls 76%, per a new arXiv paper on speculative decoding. For products that pay per cloud token, this is the most direct cost lever around.

Speculative decoding normally runs a draft model and a verifier model on one machine. SPADE splits them across devices instead. The draft model moves to the edge, and the cloud keeps the large verifier. The edge model proposes tokens locally. The cloud model checks them in parallel batches and only corrects a token when the draft model guessed wrong. Neither model needs retraining, and the method is plug-and-play, per the paper.

On SpecBench and CNN/DailyMail summarization tasks, the split cut cloud calls 76% with no drop in accuracy, the paper reports. The cloud verifier still checks every proposed token, so output quality doesn't change even as call volume drops.

Both benchmarks are structured tasks. The paper doesn't say whether the reduction holds on messier, more varied production traffic, the kind most real products actually see.