# LLMRouter

> Open-source Python library for developing, training, evaluating, and deploying LLM routers that pick the best model per query.

LLMRouter is an open-source Python library from U Lab at the University of Illinois Urbana-Champaign for building and comparing routers that choose which large language model should handle each query. It combines data generation, router training, inference, and evaluation in one workflow and ships a unified `llmrouter` command-line interface. The project is MIT licensed and installable via pip.

## What It Is

LLMRouter treats model selection as a sequential decision process: describe the query, user context, and candidate models, make a routing decision under a budget, then learn and evaluate on quality and cost. The README lists 16+ router implementations across single-round, multi-round, multimodal, agentic, and personalized categories, using methods such as KNN, SVM, MLP, matrix factorization, Elo rating, graph-based, and BERT-based routing.

## Workflow and Outputs

A data generation pipeline builds routing data from 11 benchmark datasets by calling model APIs and scoring responses. It produces query data, LLM embeddings, query embeddings, and routing train/test files with responses, performance scores, and token usage. Routers are then trained with `llmrouter train`, run with `llmrouter infer` (including a route-only mode that needs no API calls), and explored through a Gradio chat interface with current-only, full-context, and retrieval query modes. API keys can be configured per provider, and local OpenAI-compatible servers such as Ollama, vLLM, and SGLang are supported.

## xRouteBench and Evaluation

The project introduces xRouteBench, a benchmark covering generic LLM tasks, memory-augmented reasoning, vision and video, time-series, and personalized routing. A benchmark pipeline trains and evaluates 17 routers on its datasets by replaying pre-recorded model executions. The authors report that learned routers achieved a 14.6% relative improvement over the strongest fixed-model baseline.

## Extensibility and Deployment

Custom routers and custom tasks can be added through a plugin system without modifying core code. A ComfyUI interface lets users build data generation and routing pipelines visually. OpenClaw Router exposes an OpenAI-compatible server (chat completions and responses endpoints) with routing memory, media understanding, and streaming, for use with OpenClaw on platforms like Slack and Discord.

## Features
- 16+ routing models across single-round, multi-round, multimodal, agentic, and personalized categories
- Unified llmrouter CLI for train, infer, chat, serve, and list-routers
- Data generation pipeline over 11 benchmark datasets
- xRouteBench benchmark pipeline with quality–cost evaluation
- Gradio-based interactive chat interface
- Plugin system for custom routers and custom tasks
- ComfyUI visual pipeline interface
- OpenAI-compatible OpenClaw Router server with routing memory and multimodal understanding
- Support for local OpenAI-compatible LLM servers
- Multi-provider API key configuration with round-robin load balancing

## Integrations
OpenClaw, ComfyUI, Hugging Face, NVIDIA API, OpenAI, Anthropic, Together AI, Ollama, vLLM, SGLang, Slack, Discord, Gradio

## Platforms
LINUX, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Links
- Website: https://ulab-uiuc.github.io/LLMRouter/
- Documentation: https://ulab-uiuc.github.io/LLMRouter/
- Repository: https://github.com/ulab-uiuc/LLMRouter
- EveryDev.ai: https://www.everydev.ai/tools/llmrouter