# LMQL

> A Python-superset programming language for LLMs with typed variables, templates, constraints and an optimizing runtime.

LMQL is a programming language for large language models created by the SRI Lab at ETH Zurich and contributors. It is built as a superset of Python, so LLM calls are written directly inside program code rather than through a separate templating layer. It is open source under the Apache 2.0 license, and the latest release listed is v0.7.3.

## What It Is

LMQL programs read like standard Python, but top-level strings are treated as prompts sent to an LLM. Template variables such as `[ANSWER]` are completed by the model, and the results are available as normal program variables. A `where` clause attaches constraints to generated text, such as length limits, stopping phrases, set membership, or typed outputs like `int`. Query programs are functions that can be called from Python.

## How Constraints and Decoding Work

Constraints are enforced by the runtime through logit masking, which guides the model's output and intermediate results. Programs can run with several decoding algorithms, including argmax, sample, and branching decoders such as beam search and best_k. The project documents an optimizing runtime that uses speculative execution, constraint short-circuiting, and tree-based caching. Both a sync and an async API are provided, and the async API enables cross-query batching.

## Backends, Integrations and Tooling

The same LMQL code can target llama.cpp, OpenAI (including Azure OpenAI), and Hugging Face Transformers models, with the backend switched in one line. LangChain and LlamaIndex integrations are documented. A browser-based Playground IDE and a Visual Studio Code extension are available. Output can be streamed via WebSocket, REST endpoint, or Server-Sent Events.

## Newer Language Features

The homepage highlights nested queries, which allow modular local instructions and reuse of prompt components. Documented example use cases include multi-part prompts, measuring distributions, meta prompting, tool augmentation, and chatbots.

## Setup Path

Install with `pip install lmql` on Python 3.10. Local GPU models require a GPU-enabled PyTorch and `pip install lmql[hf]`. Run `lmql playground` to launch the IDE, which requires Node.js, or `lmql run` to execute local `.lmql` files. Local Hugging Face models need an LMQL inference server started with `lmql serve-model`.

## Features
- Python-superset syntax with native LLM calls
- Constraints via where clauses and logit masking
- Typed variables and regex constraints
- Nested queries
- Advanced decoding (argmax, sample, beam search, best_k)
- Optimizing runtime with speculative execution and tree-based caching
- Sync and async API with cross-query batching
- Multi-backend support (llama.cpp, OpenAI, Transformers)
- Output streaming via WebSocket, REST, SSE
- Browser-based Playground IDE
- Visual Studio Code extension

## Integrations
OpenAI, Azure OpenAI, Hugging Face Transformers, llama.cpp, LangChain, LlamaIndex, Visual Studio Code

## Platforms
WINDOWS, MACOS, WEB, API, VSC_EXTENSION, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v0.7.3

## Links
- Website: https://lmql.ai
- Documentation: https://lmql.ai/docs/
- Repository: https://github.com/eth-sri/lmql
- EveryDev.ai: https://www.everydev.ai/tools/lmql