# mistral.rs

> Fast, flexible LLM inference engine written in Rust with support for 45+ model architectures, multimodal inputs, quantization, and an OpenAI/Anthropic-compatible API server.

mistral.rs is an open-source LLM inference engine written in Rust, created by Eric Buehler and licensed under the MIT License. It supports over 45 model architectures and provides a single binary that handles interactive chat, API serving, benchmarking, and a built-in web UI. The project is actively maintained on GitHub with its latest release being v0.9.1, published in August 2026.

## What It Is

mistral.rs is a local inference runtime that lets developers and researchers run large language models on their own hardware — from Apple Silicon laptops to multi-GPU CUDA clusters — without relying on external API services. It exposes both OpenAI-compatible `/v1` endpoints and Anthropic-compatible Messages endpoints from the same `mistralrs serve` process, making it a drop-in backend for existing tooling. The engine is built on top of Hugging Face's Candle framework and is not affiliated with Mistral AI despite the name.

## Performance Architecture

The engine is engineered for throughput and latency across a range of hardware targets:

- **Continuous batching** enabled by default on all devices
- **PagedAttention** for high-throughput batching on CUDA and Apple Silicon, with prefix caching including multimodal inputs
- **FlashAttention V2/V3** on CUDA; Metal acceleration on Apple Silicon
- **Multi-GPU and distributed inference** via NCCL tensor parallelism, P2P layer mapping, multi-node, and ring configurations
- **Quantization**: ISQ (in-situ quantization), GGUF (2–8 bit), GPTQ, AWQ, HQQ, FP8, and BNB; per-layer topology control for fine-tuning quality/speed tradeoffs

The project's own v0.8.2 benchmarks show mistral.rs UQFF Q8 outperforming llama.cpp GGUF Q8_0 on prefill throughput for Gemma 4 models across GB10, B200, and H100 SXM hardware.

## Multimodal and Agentic Capabilities

mistral.rs goes beyond text-only inference with native support for vision, video, audio input, speech generation, image generation, and embeddings — all within a single engine. Its built-in agentic runtime includes:

- Web search with embedding-based result ranking
- Persistent Python code execution (Jupyter-like sessions with matplotlib capture)
- Shell execution with sandboxing and approval controls
- OpenAI-compatible Skills bundles uploaded via `/v1/skills`
- OpenAI-compatible file inputs via `/v1/files`
- MCP client support connecting to external tools via Process, HTTP, or WebSocket
- Server-side agentic loop that auto-executes tools and feeds results back to the model

## Developer SDKs and Deployment

mistral.rs ships a Python package (`pip install mistralrs`) and a Rust crate (`cargo add mistralrs`) for embedding the engine directly in applications without running a separate server. Prebuilt CPU and CUDA Docker images are published to GHCR, with Kubernetes deployment notes in the official documentation. The CLI (`mistralrs run`, `mistralrs serve`, `mistralrs bench`) auto-detects model architecture, quantization format, and chat template from Hugging Face checkpoints or local GGUF files. A `mistralrs tune` command recommends quantization and device mapping based on detected hardware.

## Update: v0.9.1

The latest release is **v0.9.1**, published on August 14, 2026. Recent additions highlighted in the README include:

- **Muse Glimmer 30B** support with native text, image, and video inference, ATEM tool calling, reasoning controls, LoRA, ISQ/UQFF, and companion-projector GGUF loading
- **DiffusionGemma** block-diffusion text generation with paged attention, prefix caching, ISQ, multimodal, and tool calling
- **Anthropic Messages API** endpoints (`/v1/messages` and `/v1/messages/count_tokens`) alongside the existing OpenAI-compatible API
- **Gemma 4** full multimodal support covering text, image, video, and audio input
- Prometheus metrics endpoint (`/metrics`) for per-request counts and latency labeled by method, route, and status

The repository has accumulated over 7,600 GitHub stars and 680 forks since its creation in February 2024, reflecting active community adoption.

## Features
- OpenAI-compatible API server
- Anthropic Messages API compatibility
- 45+ model architecture support
- GGUF model loading
- In-situ quantization (ISQ)
- GPTQ, AWQ, HQQ, FP8, BNB quantization support
- PagedAttention for high-throughput batching
- FlashAttention V2/V3 on CUDA
- Metal acceleration on Apple Silicon
- Multi-GPU and distributed inference
- Continuous batching
- Prefix caching including multimodal
- LoRA and X-LoRA adapter support
- AnyMoE mixture-of-experts
- Multimodal: text, vision, video, audio input
- Speech generation
- Image generation
- Embeddings
- Built-in web UI
- Agentic loop with tool calling
- Web search integration
- Python code execution (persistent sessions)
- Shell execution with sandboxing
- MCP client support
- OpenAI-compatible Skills bundles
- OpenAI-compatible file inputs
- Prometheus metrics endpoint
- Hardware-aware tuning (mistralrs tune)
- Python SDK
- Rust SDK
- Docker images (CPU and CUDA)
- Per-layer quantization topology control
- Auto-detection of model architecture and chat template
- Multiple model load/unload at runtime

## Integrations
Hugging Face Hub, GGUF, UQFF, GPTQ, AWQ, HQQ, BNB, Candle (Hugging Face), CUDA, Metal (Apple Silicon), NCCL, Prometheus, Docker, Kubernetes, MCP (Model Context Protocol), OpenAI API clients, Anthropic API clients, Jupyter (Python sessions)

## Platforms
WINDOWS, MACOS, LINUX, API, CLI, DEVELOPER_SDK

## Pricing
Open Source

## Version
v0.9.1

## Links
- Website: https://github.com/EricLBuehler/mistral.rs
- Documentation: https://ericlbuehler.github.io/mistral.rs/
- Repository: https://github.com/EricLBuehler/mistral.rs
- EveryDev.ai: https://www.everydev.ai/tools/mistral-rs