Edge0

Edge0 is an open-source streaming mixture-of-experts (MoE) inference framework and model family from Edge0-AI, designed to run large sparse models on consumer hardware. It can be used through a command line, an OpenAI- and Anthropic-compatible local server, or the Edge0 Chat desktop app. The latest release, dated November 2026 in the README, brings 145B and 120B latent-MoE models to a single NVIDIA GPU or an Apple Silicon Mac.

What It Is

Edge0 is an on-device inference stack. The edge0-infer program provides shared serve and chat commands and selects the engine based on the model folder. The newest 145B and 120B models run on edge0-strata, a fork of Strata, while the earlier 35B release runs on a GGML-based native engine. The models are trained natively at 4 bits and are distributed in Q3_K and NVFP4 builds for local agent workloads.

How It Runs Large Models Locally

The framework streams expert weights from SSD and keeps active memory small. Low-memory options include on-demand SSD reads (--mmap-experts), a host-RAM expert cache (--resident-budget-gib) and VRAM reservation controls. The 145B and 120B models route each token to 10 of 512 experts alongside one shared expert. The README reports benchmark throughput on specific machines, for example 121.2 tok/s agent decode for 145B Q3_K on an RTX 5090 D 24 GB.

Platforms and Setup

The 145B/120B models run on Linux or Windows with an NVIDIA GPU and CUDA 13, or on macOS with Metal on Apple Silicon. The 35B model also supports CPU, Vulkan on Windows, Metal on iOS and CPU on Android. Setup involves cloning the repository, building edge0-strata and edge0-infer, and pointing edge0-infer serve or chat at a model folder. Windows and macOS builds of Edge0 Chat are offered through GitHub Releases.

Models and Licensing

Beyond the open framework, the Edge0 website lists a family of on-device language models (Edge0-35B, Edge0-8B) and audio models (ASR and TTS) aimed at products such as earbuds, glasses, robots and toys. The framework code is Apache-2.0, while Edge0-specific model contributions use the Edge0 Community Model License 1.0, which requires separate written permission for model sales, commercial model services and independent commercial coding or office assistants.