# uzu

> An open-source Rust inference engine from Mirai for running AI models on-device on Apple Silicon, with Swift, Python, TypeScript and Rust APIs.

uzu is the inference engine from Mirai Labs, a London-founded on-device AI lab. It runs language models locally on Apple devices through a Rust core with Swift, Python and TypeScript bindings, and is released under the MIT License. The latest release listed is 0.6.15.

## What It Is

uzu is a high-performance inference engine that lets developers embed AI models directly in their apps. The README states that this gives zero latency, full data privacy and no inference costs. It uses its own model format and takes advantage of unified memory on Apple devices. Supported backends are Metal and CPU.

## How the Workflow Looks

A developer creates an Engine, picks a model by identifier, downloads it with progress updates, and opens a chat session to get replies. A loaded model can be shared across several sessions without reloading its weights. The documented examples cover:

- streaming replies
- structured JSON output constrained by a schema
- tool calls
- chatting with a cloud model through an API key

## CLI and Server

The CLI is an interactive app for browsing, downloading and chatting with models, and it can also run benchmarks. A server mode exposes an OpenAI-compatible HTTP API, with chat completions (including streaming) and model listing endpoints.

## Model Conversion and Stack Context

Models can be converted for uzu with the separate lalamo tool. Mirai describes uzu as one layer of a stack that also includes its own models and quantization, designed around batch-size-1, memory-bound on-device execution on Apple Silicon. Mirai publishes benchmarks comparing uzu with other engines such as MLX and llama.cpp. These are vendor-published results, and the Android inference runtime is listed as coming soon.

## Features
- On-device inference on Apple Silicon using Metal and CPU backends
- Rust core with Swift, Python and TypeScript bindings
- Model download with progress updates
- Streaming chat replies
- Structured JSON output via JSON schema grammar
- Tool calling
- Shared chat instances to reuse loaded model weights
- Optional cloud model chat via API key
- Interactive CLI with model browsing and benchmarking
- OpenAI-compatible HTTP server
- Model conversion and quantization with lalamo

## Integrations
Swift Package Manager, PyPI, npm, Hugging Face, OpenAI API, lalamo

## Platforms
MACOS, LINUX, ANDROID, IOS, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
0.6.15

## Links
- Website: https://trymirai.com
- Documentation: https://docs.trymirai.com
- Repository: https://github.com/trymirai/uzu
- EveryDev.ai: https://www.everydev.ai/tools/uzu