Tools
A Vulkan shader constant capped expert hoisting at 256, costing Qwen3.8-Flash-Next 19% of its prefill
llama.cpp PR #28501 (merged 2026-09-18 07:00 UTC) found `count_experts.comp` sized its shared arrays with `BLOCK_SIZE` (256), so `ggml_vk_mul_mat_id_q_f16` disabled row-id hoisting entirely for models with more than 256 experts and every workgroup rescanned the full ids tensor. Qwen3.8-Flash-Next has 512 experts. Raising the limit to a separate `MAX_EXPERTS` of 512 took per-op time from 14,033µs to 7,508µs on iq4_nl 2560x640 (Strix Halo, Radeon 8060S, RADV), and Q5_K prefill from 426 to 507 t/s at 8k (+19%) and 333 to 387 t/s at 32k (+16%), with token generation unchanged and greedy output identical.
Source
↳ Follow the thread