LLMRouter

LLMRouter is an open-source Python library from U Lab at the University of Illinois Urbana-Champaign for building and comparing routers that choose which large language model should handle each query. It combines data generation, router training, inference, and evaluation in one workflow and ships a unified llmrouter command-line interface. The project is MIT licensed and installable via pip.

What It Is

LLMRouter treats model selection as a sequential decision process: describe the query, user context, and candidate models, make a routing decision under a budget, then learn and evaluate on quality and cost. The README lists 16+ router implementations across single-round, multi-round, multimodal, agentic, and personalized categories, using methods such as KNN, SVM, MLP, matrix factorization, Elo rating, graph-based, and BERT-based routing.

Workflow and Outputs

A data generation pipeline builds routing data from 11 benchmark datasets by calling model APIs and scoring responses. It produces query data, LLM embeddings, query embeddings, and routing train/test files with responses, performance scores, and token usage. Routers are then trained with llmrouter train, run with llmrouter infer (including a route-only mode that needs no API calls), and explored through a Gradio chat interface with current-only, full-context, and retrieval query modes. API keys can be configured per provider, and local OpenAI-compatible servers such as Ollama, vLLM, and SGLang are supported.

xRouteBench and Evaluation

The project introduces xRouteBench, a benchmark covering generic LLM tasks, memory-augmented reasoning, vision and video, time-series, and personalized routing. A benchmark pipeline trains and evaluates 17 routers on its datasets by replaying pre-recorded model executions. The authors report that learned routers achieved a 14.6% relative improvement over the strongest fixed-model baseline.

Extensibility and Deployment

Custom routers and custom tasks can be added through a plugin system without modifying core code. A ComfyUI interface lets users build data generation and routing pipelines visually. OpenClaw Router exposes an OpenAI-compatible server (chat completions and responses endpoints) with routing memory, media understanding, and streaming, for use with OpenClaw on platforms like Slack and Discord.