XGrammar

XGrammar is an open-source library developed under the mlc-ai organization that brings fast, flexible structured generation to large language model inference. Released under the Apache 2.0 license, it is installable via pip and integrates directly into LLM inference engines as a backend for constrained decoding. The project publishes versioned releases on GitHub and PyPI, with active development continuing through 2026.

What It Is

XGrammar is a structured generation engine that uses constrained decoding to guarantee 100% structural correctness in LLM outputs. It supports general context-free grammar, enabling a broad range of output formats including JSON, regex, and custom EBNF or Lark grammars. The library is built around a minimal, portable C++ backend that can be embedded into diverse runtime environments, and it exposes APIs for Python, C++, JavaScript, and Swift. The project's stated mission is to bring "flexible zero-overhead structure generation everywhere."

How Constrained Decoding Works

XGrammar intercepts the token generation process and applies grammar-based bitmask operations to filter the model's logit distribution at each decoding step, ensuring only tokens that conform to the target structure are sampled. Key components in the API include GrammarCompiler, CompiledGrammar, and GrammarMatcher, which together compile a grammar definition and then match it token-by-token during inference. The library also supports a "Structural Tag" system for tool calling and reasoning workflows, allowing structured outputs to be interleaved with free-form text.

Platform and Hardware Coverage

The library is designed for universal deployment across a wide range of environments:

• Operating systems: Linux, macOS, Windows
• Hardware: CPU, NVIDIA GPU, AMD GPU, Apple Silicon (MPS), TPU
• Languages: Python, C++, JavaScript, Swift
• Models: Qwen, Llama, DeepSeek, Phi, Gemma, and others

For Apple Silicon users, a Metal-accelerated variant is available via pip install "xgrammar[metal]". Community Rust bindings are also available through the third-party xgrammar-rs project.

LLM Engine Integration

XGrammar is the default structured generation backend for several major open-source LLM inference engines. According to the project's README, it has been officially integrated into vLLM (December 2024), SGLang (November 2024), TensorRT-LLM (January 2025), MLC-LLM (November 2024), Modular's MAX (February 2025), OpenVINO GenAI (September 2025), and Mirai (December 2025). The library is co-designed with LLM inference engines to enable what the project describes as "zero-overhead structured generation."

Update: XGrammar-2 and v0.2.8

In May 2026, the team released XGrammar-2, described in a blog post as a "fast, customizable structured generation" engine targeting agentic LLM workloads. A corresponding paper, "XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMs," was published in the Proceedings of the ACM Conference on AI and Agentic Systems (2026). The latest stable release on GitHub is v0.2.8, published September 24, 2026, with a development version (0.2.9.dev) visible in the documentation. The project maintains an active release cadence without a fixed calendar schedule, publishing as features and fixes are ready.