TokenSpeed

TokenSpeed is an open-source LLM inference engine from the LightSeek Foundation, a 501(c)(3) nonprofit. It targets agentic workloads such as coding agents, which involve long contexts and many conversation turns. The project is MIT-licensed, and its latest listed release is v0.1.1.

What It Is

TokenSpeed is a serving engine for large language models. The README describes the goal as TensorRT-LLM-level performance with vLLM-level usability. Its core components are a scheduler, a layered kernel system, and an SMG-integrated AsyncLLM entrypoint for low-overhead CPU-side request handling.

How the Architecture Works

The scheduler separates the control plane from the execution plane. The control plane is a C++ finite-state machine that encodes request lifecycle, KV cache ownership, and overlap timing, and the type system enforces safe KV resource reuse at compile time. The execution plane is written in Python for faster iteration.

The modeling layer uses a local SPMD design. Developers annotate I/O placement at module boundaries, and a lightweight static compiler generates the required collective operations during model construction, so communication logic does not have to be written by hand.

Kernels are a modular subsystem with a portable public API, a centralized registry and selection model, and a plugin mechanism for heterogeneous accelerators. This includes TokenSpeed MLA, a Multi-head Latent Attention kernel optimized for NVIDIA Blackwell.

Performance Claims

The project's launch blog reports benchmarks against TensorRT-LLM on Kimi K2.5 on NVIDIA B200, using SWE-smith traces with EvalScope. For the Attention TP4 + MoE TP4 configuration it reports roughly 9% faster minimum latency and roughly 11% higher throughput around 100 TPS/User. Those figures are the team's own results and cover non-disaggregated deployment.

Current Status

The launch blog says the engine and kernels were under active development, with production hardening planned. The README news lists day-0 model support, AMD and NVIDIA platform work, and inclusion in the PyTorch Ecosystem Landscape.