vLLM Semantic Router

vLLM Semantic Router is an open-source, signal-driven routing framework for heterogeneous LLM inference, developed under the vllm-project organization and licensed under Apache 2.0. It enables teams to build Mixture-of-Models systems by evaluating request signals, user preferences, and application policies to select or compose the right model path for each request — without hard-coding routing logic into applications. The project is written primarily in Go, has accumulated over 5,800 GitHub stars, and is actively maintained by contributors from AMD, Microsoft, Red Hat, Delta Electronics, and other organizations.

What It Is

vLLM Semantic Router sits as a programmable routing layer between client applications and a heterogeneous pool of LLM backends — including closed models (Claude, ChatGPT, Gemini), open models (Mistral, DeepSeek, Llama), and hybrid configurations. Rather than sending every request to a single model, it evaluates up to 20 signal families — spanning domain, PII, jailbreak detection, embedding similarity, complexity, history, tool use, and more — then applies a configured recipe policy to select or compose the best model path. The result is a single OpenAI-compatible API surface that abstracts the entire model fleet.

Architecture: Signal → Decision → Model Path

The router maps each incoming request through a four-layer pipeline inspired by Shannon's communication theory:

• Signal Extraction: 20 signal families covering request context, safety, intent, preference, and system state
• Decision Engine: 11 selection algorithms (e.g., router_dc for embedding-similarity matching, multi_factor for balancing quality/latency/cost/load) plus 5 loopers for cascading, escalating, or fusing across models
• Policy & Guardrails: Inline PII and jailbreak classifiers, authorization-aware routing, per-route rate limits, and hard constraints for data residency and locality — ineligible paths are removed before ranking runs
• Observability: OpenTelemetry-ready metrics export, replay-enabled routing records with x-vsr-replay-id response headers, and dashboard analytics

Deployment Options

The router supports four primary deployment paths:

• Envoy ExtProc — production gateway filter
• Gateway API — Agentgateway integration for Kubernetes
• K8s Operator — declarative fleet routing
• Local vllm-sr — single-line install for dev and laptop workflows (curl -fsSL https://vllm-sr.ai/install.sh | bash -s -- --channel stable)

An online playground is available at app.vllm-sr.ai for hands-on exploration without local setup.

Research Foundation

The project publishes actively across routing, safety, multimodality, and systems design. Notable papers include the position paper introducing the signal-driven decision routing framework for Mixture-of-Modality deployments (arXiv 2603.04444), the Workload-Router-Pool (WRP) vision paper (arXiv 2603.21354), and work on 98× faster routing latency using Flash Attention and prompt compression (arXiv 2603.12646). Additional papers cover span-level hallucination detection, adaptive VLM routing for computer-use agents, token-budget-aware pool routing, and IETF drafts for the Semantic Inference Routing Protocol (SIRP). The project lists 19 research papers in total across its publications page.

Update: v0.3 Themis Release

The latest stable release is v0.3.0 "Themis", published on June 5, 2026, described by the project blog as moving "from signals to stateful production routing." Prior major releases include v0.2 "Athena" (March 2026) and v0.1 "Iris" (January 2026). A July 2026 blog post announced a new chapter focused on Mixture-of-Models systems. The project also publishes a Micro-Agent capability (June 2026) enabling router-side model collaboration that the project claims can match or exceed frontier model benchmarks on LiveCodeBench (92.6) and GPQA-Diamond (96.0) by coordinating closed and open model pools behind a single API call.