BaseRT

BaseRT is an open-source LLM inference runtime built by Base Compute, an AI inference lab with teams in Melbourne and Berlin. It targets Apple Silicon (M1 or later, macOS 14+) and delivers benchmark-leading throughput through hand-written Metal GPU kernels. The project repository is licensed under Apache-2.0, while the prebuilt engine binary ships under its own separate proprietary license.

What It Is

BaseRT is a local inference engine that lets developers pull models from HuggingFace, chat with them interactively, or serve an OpenAI-compatible HTTP API — all from a single CLI. It introduces its own .base model format with affine quantization (Q2–Q8), optional AWQ calibration, and signed bundles. The runtime supports text, vision, and audio on compatible models, and exposes a stable C API with bindings for Python, Node, Rust, and Swift.

Performance on Apple Silicon

The BaseRT homepage publishes benchmark results measured on an Apple M5 Pro. According to those figures:

• Prefill: up to 6.4x faster than llama.cpp and 3.9x faster than MLX (e.g., Gemma 4 E2B at pp2048 Q8: 16,264 tokens/sec vs. 2,547 for llama.cpp)
• Decode: up to 1.33x faster than MLX (e.g., Qwen3 0.6B Q4: 531 tokens/sec vs. 398 for MLX)

These gains come from hand-tuned Metal kernels optimized per hardware and per model architecture.

CLI and Workflow

The basert CLI is the single entry point for all operations:

• basert pull <id> — download from HuggingFace and convert to .base
• basert chat <model> — interactive chat session
• basert serve <model> — OpenAI-compatible HTTP server with continuous batching, paged-KV, and prefix caching
• basert convert <src> — convert GGUF, HF, or MLX checkpoints to .base
• basert bench <model> — throughput benchmark
• basert sign / verify / keygen — ed25519 signing of .base bundles

The server supports chat completions, embeddings, transcription, and tool calls out of the box.

Supported Models and Coding Agent Integration

BaseRT supports a broad set of popular open-weight model families including Qwen3, Qwen3.5, Qwen3.6, Llama 3.1, Llama 3.2, Gemma 3, Gemma 4, Mistral, Phi-3, and Nomic BERT. It also functions as a drop-in local backend for the pi coding agent via the pi-basert extension, which auto-discovers running models. This allows developers to run a full local coding agent workflow — model serving, agent execution, and tool calling — entirely on-device with no API keys and no data leaving the machine.

Update: BaseRT Engine v0.1.7

The latest release is BaseRT engine 0.1.7, published on July 21, 2026. The GitHub repository was created in late June 2026 and has seen active development, with the last push on July 21, 2026. The project is in early versioning (0.x) and the repository currently has 88 stars and 6 forks. Base Compute describes its research direction as focused on GPU kernel optimization and automated research pipelines combining human and AI collaboration.

Open-Source Licensing and Repository Structure

The repository is organized into distinct components: base-convert/ (the Rust CLI, model hub, converter, and .base format specs), include/baseRT/ (the public C API headers), bindings/ (Python, Node, Rust, Swift), benchmarks/, and docs/. The CLI, .base format, public headers, and language bindings are all Apache-2.0 licensed. The prebuilt engine binary is distributed separately under a proprietary license and is not open source.