XGrammar: 100× Faster Constrained Decoding Now Default in vLLM—Grammar Caching Is the Key Production Optimization

MLC AI Blog·high signal

XGrammar enforces JSON Schema and Pydantic models against LLM token logits at inference time at under 40 microseconds per token, and is the default structured output backend in vLLM 0.4+ and SGLang—replacing prior regex approaches by 100×. The non-obvious production optimization is reusing compiled grammar objects across requests rather than re-instantiating per call; XGrammar caches compiled FSMs and near-zero overhead only holds when cache hit rates are high. Microsoft's llguidance (Rust-based, also credited by OpenAI for their Structured Outputs) achieves ~50μs/token with negligible startup cost and is the alternative for non-vLLM deployments.