transformers v5.16.0 adds Qwen4-Exp, the first hybrid combining linear and sparse attention.

Three new components ship with it: GatedResidual, which mixes multiple residual streams with elementwise gating before each attention and MoE block; Qwen Sparse Attention, which scores compressed key blocks and selects contiguous token blocks while leaving the trailing block uncompressed; and Per-Layer Embedding, enriching selected decoder layers with hashed token n-gram features (GitHub). Combined with Gated DeltaNet, QSA makes Qwen4-Exp the first architecture integrating both linear and sparse attention. The block-level selection targets long-context inference cost, which is the binding constraint on agent workloads.