RouteLLM

RouteLLM is an open-source Python framework from the lm-sys organization for serving and evaluating LLM routers. It routes each query between a stronger, more expensive model and a cheaper, weaker one, and can be used as a drop-in replacement for OpenAI's client or as an OpenAI-compatible server.

What It Is

RouteLLM addresses the cost-quality tradeoff in deploying LLMs: sending everything to the most capable model is costly, while smaller models may reduce quality. A router inspects each query and decides which of two models should handle it. Each request carries a cost threshold; a higher threshold lowers cost but may lower response quality. The project's authors say the provided routers reduced costs by up to 85% while keeping 95% of GPT-4 performance on benchmarks such as MT Bench, and that they match commercial offerings while being over 40% cheaper.

How Routing Works

Users initialize a Controller with a router, a strong model and a weak model, then set the model field to router-[ROUTER NAME]-[THRESHOLD], for example router-mf-0.11593. A calibration command determines the threshold that sends a chosen percentage of queries to the strong model, using the public Chatbot Arena dataset by default. The same routing is available through a lightweight OpenAI-compatible server, and a local router chatbot demo shows how messages are routed.

Included routers are:

• mf: matrix factorization, the recommended one
• sw_ranking: weighted Elo
• bert: BERT classifier
• causal_llm: LLM-based classifier
• random

The trained routers were built on the gpt-4-1106-preview and mixtral-8x7b-instruct-v0.1 pair, and the authors report they generalize to other model pairs without retraining.

Model Support and Evaluation

RouteLLM uses LiteLLM to reach many open and closed models, including Anthropic, Gemini, Amazon Bedrock, Together AI, Anyscale and local models via Ollama, plus any OpenAI-compatible endpoint. An OpenAI API key is still required for the embeddings used by the mf and sw_ranking routers.

An evaluation framework compares routers on the MMLU, GSM8K and MT Bench benchmarks, prints results and generates a performance plot. New routers can be added by implementing the abstract Router class, and new benchmarks by implementing the Benchmark class. The project is released under Apache License 2.0, and the README links a paper and blog post describing the research.