Tools
llama.cpp fuses top-k MoE routing on Metal for up to 1.16x token generation on M2 Ultra
PR #28948, merged 2026-09-19, adds four Metal fusions: SOFT_MAX + ARGSORT + GET_ROWS for top-k MoE routing with optional normalization and scaling, a MUL + expert-views + ADD weighted reduction, RMS_NORM + SCALE, and SSM_CONV + silu applied through function constants. Measured on M2 Ultra across DeepSeek-4B MoE, Gemma-26B, Qwen3.5 0.8B and 27B and Qwen3.5 MoE-35B, speedups run 1.00x to 1.16x, with Qwen MoE at 1.05x prompt processing and 1.16x token generation. M5 Max lands lower at 1.00x to 1.12x, Qwen MoE hitting 1.07x pp2048 and 1.12x tg128.
Source
↳ Follow the thread