uzu

uzu is the inference engine from Mirai Labs, a London-founded on-device AI lab. It runs language models locally on Apple devices through a Rust core with Swift, Python and TypeScript bindings, and is released under the MIT License. The latest release listed is 0.6.15.

What It Is

uzu is a high-performance inference engine that lets developers embed AI models directly in their apps. The README states that this gives zero latency, full data privacy and no inference costs. It uses its own model format and takes advantage of unified memory on Apple devices. Supported backends are Metal and CPU.

How the Workflow Looks

A developer creates an Engine, picks a model by identifier, downloads it with progress updates, and opens a chat session to get replies. A loaded model can be shared across several sessions without reloading its weights. The documented examples cover:

• streaming replies
• structured JSON output constrained by a schema
• tool calls
• chatting with a cloud model through an API key

CLI and Server

The CLI is an interactive app for browsing, downloading and chatting with models, and it can also run benchmarks. A server mode exposes an OpenAI-compatible HTTP API, with chat completions (including streaming) and model listing endpoints.

Model Conversion and Stack Context

Models can be converted for uzu with the separate lalamo tool. Mirai describes uzu as one layer of a stack that also includes its own models and quantization, designed around batch-size-1, memory-bound on-device execution on Apple Silicon. Mirai publishes benchmarks comparing uzu with other engines such as MLX and llama.cpp. These are vendor-published results, and the Android inference runtime is listed as coming soon.