# NVIDIA LLM Router

> An open-source NVIDIA AI Blueprint that intelligently routes LLM requests to the optimal model based on task complexity, intent, and cost-quality-latency tradeoffs.

NVIDIA LLM Router is an open-source AI Blueprint published by NVIDIA under the Apache 2.0 license, designed to automatically route LLM and Vision-Language Model (VLM) requests to the most appropriate model for a given task. It is hosted on GitHub under the NVIDIA-AI-Blueprints organization and is accessible as a developer example on NVIDIA's build.nvidia.com platform. The project page notes that this Blueprint has been deprecated, with NVIDIA directing users to transition to NVIDIA NeMo Switchyard, the successor routing library.

## What It Is

LLM Router is a model-routing middleware that sits between an application and a pool of LLMs or VLMs. Rather than always calling the most powerful (and expensive) frontier model, it analyzes each incoming prompt — including text and images — and returns the name of the most appropriate model to handle that request. The router itself does not proxy calls to downstream models in v2; it returns a model recommendation via an OpenAI-compatible chat completions endpoint, and the calling application is responsible for making the actual inference call.

## Two Routing Strategies

The v2 (experimental) router offers two distinct approaches:

- **Intent-Based Routing**: Uses a small Qwen 1.7B LLM to classify the user's intent (e.g., `chit_chat`, `hard_question`, `image_understanding`) and maps each intent to a configured model. No training is required; intent mappings are defined in a YAML config file.
- **Auto-Routing (CLIP + Neural Network)**: Uses CLIP embeddings to encode text/image pairs, then passes them through a trained neural network that predicts the optimal model based on quality, latency, and cost thresholds. This approach learns from historical usage patterns and can be retrained on custom data using provided Jupyter notebooks.

## Architecture and Components

The v2 blueprint is structured around three main components:

- **Router Backend**: A FastAPI service built on the NVIDIA NeMo Agent Toolkit, exposing an OpenAI-compatible endpoint at `/sfc_router/chat/completions`. It accepts multimodal inputs (text and base64-encoded images) and returns the recommended model name.
- **Routing Models**: Either the Qwen 1.7B intent classifier (running on port 8011) or a CLIP embedding server (port 51000) paired with a pre-trained neural network stored in the repository.
- **Demo Application**: An interactive Gradio web interface (port 7860) that demonstrates end-to-end routing and model calling, including multimodal support.

Deployment is handled via Docker Compose with two profiles: `intent` for the intent-based router and `nn` for the neural network router.

## Hardware and Software Requirements

The Qwen 1.7B intent router requires a GPU with at least 16 GB of memory (T4 or newer). The CLIP embedding server for auto-routing requires 8 GB+ of GPU memory. Neural network training can run on CPU but benefits from GPU acceleration. The software stack requires Linux (Ubuntu 22.04+) or macOS, Docker, Docker Compose, and Python 3.12+ for local development.

## Deprecation and Successor

The GitHub README prominently marks this project as deprecated. NVIDIA announced NVIDIA NeMo Switchyard as the replacement, described as implementing the same routing patterns as LLM Router v1 and v2 while adding broader extensibility, improved performance, and deeper integrations with the NVIDIA NeMo Platform. The build.nvidia.com project page also displays a deprecation notice advising users to transition to avoid service interruptions. The repository remains active with the latest push recorded in July 2026 and was last updated in September 2026, suggesting ongoing maintenance during the transition period. The latest formal release is v1.0, published in February 2025.

## Features
- Intent-based LLM routing using Qwen 1.7B
- Auto-routing via CLIP embeddings and trained neural network
- Multimodal support (text and images)
- OpenAI-compatible chat completions API
- Docker Compose deployment with profile switching
- Pre-trained neural network router artifacts included
- Jupyter notebooks for training and evaluation
- Interactive Gradio demo application
- Configurable intent-to-model mappings
- Support for frontier and open models
- Cost, quality, and latency threshold configuration

## Integrations
NVIDIA NeMo Agent Toolkit, NVIDIA NIM, NVIDIA Build API, Azure OpenAI, OpenAI API, Docker, Docker Compose, FastAPI, Gradio, Qwen 1.7B, NVIDIA NVClip, NVIDIA Triton Inference Server, Jupyter Lab

## Platforms
MACOS, LINUX, WEB, API, CLI

## Pricing
Open Source

## Version
v1.0

## Links
- Website: https://build.nvidia.com/nvidia/llm-router
- Documentation: https://github.com/NVIDIA-AI-Blueprints/llm-router
- Repository: https://github.com/NVIDIA-AI-Blueprints/llm-router
- EveryDev.ai: https://www.everydev.ai/tools/nvidia-llm-router