NVIDIA LLM Router

NVIDIA LLM Router is an open-source AI Blueprint published by NVIDIA under the Apache 2.0 license, designed to automatically route LLM and Vision-Language Model (VLM) requests to the most appropriate model for a given task. It is hosted on GitHub under the NVIDIA-AI-Blueprints organization and is accessible as a developer example on NVIDIA's build.nvidia.com platform. The project page notes that this Blueprint has been deprecated, with NVIDIA directing users to transition to NVIDIA NeMo Switchyard, the successor routing library.

What It Is

LLM Router is a model-routing middleware that sits between an application and a pool of LLMs or VLMs. Rather than always calling the most powerful (and expensive) frontier model, it analyzes each incoming prompt — including text and images — and returns the name of the most appropriate model to handle that request. The router itself does not proxy calls to downstream models in v2; it returns a model recommendation via an OpenAI-compatible chat completions endpoint, and the calling application is responsible for making the actual inference call.

Two Routing Strategies

The v2 (experimental) router offers two distinct approaches:

• Intent-Based Routing: Uses a small Qwen 1.7B LLM to classify the user's intent (e.g., chit_chat, hard_question, image_understanding) and maps each intent to a configured model. No training is required; intent mappings are defined in a YAML config file.
• Auto-Routing (CLIP + Neural Network): Uses CLIP embeddings to encode text/image pairs, then passes them through a trained neural network that predicts the optimal model based on quality, latency, and cost thresholds. This approach learns from historical usage patterns and can be retrained on custom data using provided Jupyter notebooks.

Architecture and Components

The v2 blueprint is structured around three main components:

• Router Backend: A FastAPI service built on the NVIDIA NeMo Agent Toolkit, exposing an OpenAI-compatible endpoint at /sfc_router/chat/completions. It accepts multimodal inputs (text and base64-encoded images) and returns the recommended model name.
• Routing Models: Either the Qwen 1.7B intent classifier (running on port 8011) or a CLIP embedding server (port 51000) paired with a pre-trained neural network stored in the repository.
• Demo Application: An interactive Gradio web interface (port 7860) that demonstrates end-to-end routing and model calling, including multimodal support.

Deployment is handled via Docker Compose with two profiles: intent for the intent-based router and nn for the neural network router.

Hardware and Software Requirements

The Qwen 1.7B intent router requires a GPU with at least 16 GB of memory (T4 or newer). The CLIP embedding server for auto-routing requires 8 GB+ of GPU memory. Neural network training can run on CPU but benefits from GPU acceleration. The software stack requires Linux (Ubuntu 22.04+) or macOS, Docker, Docker Compose, and Python 3.12+ for local development.

Deprecation and Successor

The GitHub README prominently marks this project as deprecated. NVIDIA announced NVIDIA NeMo Switchyard as the replacement, described as implementing the same routing patterns as LLM Router v1 and v2 while adding broader extensibility, improved performance, and deeper integrations with the NVIDIA NeMo Platform. The build.nvidia.com project page also displays a deprecation notice advising users to transition to avoid service interruptions. The repository remains active with the latest push recorded in July 2026 and was last updated in September 2026, suggesting ongoing maintenance during the transition period. The latest formal release is v1.0, published in February 2025.