LMQL

LMQL is a programming language for large language models created by the SRI Lab at ETH Zurich and contributors. It is built as a superset of Python, so LLM calls are written directly inside program code rather than through a separate templating layer. It is open source under the Apache 2.0 license, and the latest release listed is v0.7.3.

What It Is

LMQL programs read like standard Python, but top-level strings are treated as prompts sent to an LLM. Template variables such as [ANSWER] are completed by the model, and the results are available as normal program variables. A where clause attaches constraints to generated text, such as length limits, stopping phrases, set membership, or typed outputs like int. Query programs are functions that can be called from Python.

How Constraints and Decoding Work

Constraints are enforced by the runtime through logit masking, which guides the model's output and intermediate results. Programs can run with several decoding algorithms, including argmax, sample, and branching decoders such as beam search and best_k. The project documents an optimizing runtime that uses speculative execution, constraint short-circuiting, and tree-based caching. Both a sync and an async API are provided, and the async API enables cross-query batching.

Backends, Integrations and Tooling

The same LMQL code can target llama.cpp, OpenAI (including Azure OpenAI), and Hugging Face Transformers models, with the backend switched in one line. LangChain and LlamaIndex integrations are documented. A browser-based Playground IDE and a Visual Studio Code extension are available. Output can be streamed via WebSocket, REST endpoint, or Server-Sent Events.

Newer Language Features

The homepage highlights nested queries, which allow modular local instructions and reuse of prompt components. Documented example use cases include multi-part prompts, measuring distributions, meta prompting, tool augmentation, and chatbots.

Setup Path

Install with pip install lmql on Python 3.10. Local GPU models require a GPU-enabled PyTorch and pip install lmql[hf]. Run lmql playground to launch the IDE, which requires Node.js, or lmql run to execute local .lmql files. Local Hugging Face models need an LMQL inference server started with lmql serve-model.