colibri

Colibrì is an open-source inference engine written in pure C with zero runtime dependencies, released under Apache 2.0. It enables frontier-scale Mixture-of-Experts (MoE) models — from 35B to 2.8 trillion parameters — to run on consumer and heterogeneous hardware by treating VRAM, RAM, and NVMe storage as a single unified inference hierarchy. The project started as a one-person effort on a 12-core laptop with 25 GB of RAM and has grown into a community-driven research platform.

What It Is

Colibrì is a local inference engine and open research platform targeting the systems boundary between model formats, memory hierarchy, storage I/O, placement, scheduling, kernels, and CPU/GPU overlap. Its core insight is that a 744B MoE model activates only ~40B parameters per token, and only ~11 GB of those change token-to-token (the routed experts). Rather than requiring the full model to fit in fast memory, colibrì keeps dense weights resident and streams the 19,456 routed experts from disk on demand, using a per-layer LRU cache, a learned pinned hot-store, and one-layer-ahead prefetch to hide latency. The engine is deliberately small: one .c file per model family, shared single-header utilities, and no BLAS or Python at runtime.

Supported Model Families

Six model families run on the same engine today, each with its own architecture-specific C file but a shared coli chat / coli serve / coli web front end:

• GLM-5.2 — 744B MoE, the reference target; int4 container, token-exact vs. reference
• Kimi K3 — 2.8T MoE (Moonshot AI); gated recurrence + MLA attention, Metal backend on Apple Silicon
• Inkling — 975B MoE (Thinking Machines); audio tower, mixed-precision staging
• DeepSeek V4 Flash — 284B MoE; sparse attention, multi-token prediction, native fp4 experts
• Qwen3.6 — 35B MoE (Alibaba); Gated Attention + Gated DeltaNet, CUDA VRAM expert tier
• OLMoE — 7B MoE (Allen AI); small research workhorse for quantization and quality ablations

Memory Hierarchy and Core Techniques

The engine treats VRAM, RAM, and NVMe as placement tiers for the same weights. Insufficient fast memory changes speed, never model semantics. Key techniques include:

• JIT-style weight staging: measured routing heat drives a per-layer LRU, a learned pinned hot-store, and one-layer-ahead prefetch (routing is measured at 71.6% predictable one layer ahead)
• Batch-union reads: each unique expert is read once per token batch; three matrices stored adjacently are fetched in a single pread
• Dual-SSD striping: a second copy of the model on a second drive doubles read bandwidth; a 9 GB/s + 3 GB/s pair reads experts ~33% faster than the fast drive alone
• Heterogeneous backends: CPU, CUDA, Metal (Apple Silicon), and Vulkan (any GPU with Vulkan 1.2, including AMD via Mesa/RADV) share one runtime
• 57× smaller KV state: MLA attention stores 576 floats/token instead of 32,768, persisted across restarts for zero re-prefill on conversation resumption
• Speculative decoding: native MTP head drafts 2.2–2.8 tokens/forward when it pays; grammar-forced drafts add near-free acceptance on constrained JSON output

Hardware Range and Measured Performance

The same engine and int4 container span a wide hardware range, with the hardware only changing where experts live. Community-measured decode speeds from the benchmark tables include:

• 6× RTX 5090 with full expert residency: 5.8–6.8 tok/s decode
• AWS Graviton4 r8g.16xlarge (64-core, 512 GB RAM, CPU-only): 8.0 tok/s
• 2× Xeon Gold 6430, 1 TB DDR5 (CPU-only, all experts pinned): 5.42 tok/s
• MacBook Pro M5 Max, 128 GB, Metal: 2.0 tok/s
• Ryzen 9 9950X3D + RTX 5090 + Gen5 NVMe: 1.23 tok/s
• 25 GB dev box, cold (streamed from NVMe): 0.05–0.1 tok/s — the proven floor

Update: v1.8.0

The latest release is v1.8.0, published on 2026-08-24. The GitHub repository shows active development with 26,178 stars, 2,866 forks, and over 100 contributors as of the data collected. The project reached this state in approximately seven weeks of open development. Recent additions include the Vulkan backend (covering AMD cards via Mesa/RADV, including cards ROCm no longer supports), the DeepSeek V4 Flash engine with native fp4 expert streaming, dual-SSD striping, local cluster mode for distributing expert execution across machines over TCP, and the expert atlas — a 3-D visualization of 13,260 characterized experts mapped by measured routing affinity.

Open Research Model and Setup Path

Colibrì is explicitly positioned as an open research platform where optimizations are treated as hypotheses until controlled end-to-end A/B measurements confirm them. Active open hypotheses include routing-history-based expert placement vs. plain LRU, dual-SSD bandwidth aggregation, hardware-aware automatic planning, and routing-aware speculation break-even surfaces.

Getting started requires two things: the engine binary (a few hundred KB, available as prebuilt releases for Linux, macOS, and Windows) and a model checkpoint (e.g., ~372 GB for GLM-5.2 int4). No GPU is required for any model family. The coli launcher reads config.json and selects the correct engine binary automatically, so the same command line works across all six model families.