FreeToken

FreeToken is an open-source, edge-native inference engine built by FlashML-org that enables running frontier-scale Mixture-of-Experts (MoE) models on personal consumer hardware — including gaming desktops, laptops, and workstations — without requiring datacenter infrastructure. It is licensed under Apache 2.0 and available as both a desktop application and a Python CLI package.

What It Is

FreeToken is a MoE serving engine that treats heterogeneous edge resources — GPUs, CPUs, host memory, and interconnects — as a unified, elastic inference platform. Rather than requiring dedicated server hardware, it enables users to run models like DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2 at interactive speeds on NVIDIA RTX 30, 40, and 50 series consumer GPUs. The project is backed by a research paper (arXiv:2608.16157) titled "FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution."

Core Technical Architecture

FreeToken's design centers on several key systems that work together to maximize inference throughput on memory-constrained consumer hardware:

• Bandwidth-Adaptive CPU–GPU Co-execution (the q★ policy): Dynamically balances computation between CPU and GPU based on available memory bandwidth.
• Full-Layer Double-Buffered Prefill Streaming: Overlaps data transfer and computation during the prefill phase to reduce latency.
• Global LRU Expert Caching: Caches frequently used MoE experts in VRAM using a least-recently-used eviction policy to minimize redundant weight loading.
• Semantic Anchor Checkpoints: Caches recurrent state and KV caches at semantic boundaries, allowing agentic workflows (tool calls, thinking blocks) to skip redundant context recomputation.
• Elastic VRAM Management: Dynamically reallocates VRAM between expert caches and KV memory at runtime without engine restarts or weight reloading.
• FTW Fast Weight Format: A custom weight format optimized for edge inference throughput.

Supported Models and Quantization

FreeToken supports frontier open-weight MoE models across various parameter scales and quantization formats:

• Models: DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2, and others
• Quantization: MXFP4, NVFP4, FP8, BF16
• Hardware: NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs; scales from consumer laptops to workstations

The engine exposes Anthropic/OpenAI-compatible APIs, enabling drop-in integration with coding agents and tool-calling workflows such as Codex, Claude Code, OpenCode, OpenClaw, and DeepSeek Harness.

Setup Path

FreeToken offers two primary access methods:

• Desktop App: A GUI application for Windows and Linux, downloadable from flashml.ai, that handles engine setup and provides a chat interface and engine tuning controls.
• CLI / Python Package: Installable via uv pip install "freetoken[accel]" or pip, with full source build support. Documentation covers installation, quickstart, supported models, and CLI reference.

A developer Slack workspace and community Discord are available for support and discussion.

Update: v0.1.2

The latest release is v0.1.2, published on August 19, 2026. The repository was created in July 2026 and has seen active development, with the last push recorded on August 20, 2026. The project has accumulated over 2,200 GitHub stars and nearly 200 forks since its creation, reflecting rapid early community interest. The research paper underpinning the engine is available on arXiv (arXiv:2608.16157). The project acknowledges design inspiration and code reuse from SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM, and llama.cpp.