NVIDIA NeMo Switchyard

Switchyard is an open-source routing library from NVIDIA's NeMo organization that helps an AI agent choose which model handles each request. It combines efficient models with more capable ones so teams can balance task success, cost, and latency. The latest release listed is v0.3.0, and the README describes the project as pre-1.0 software.

What It Is

Switchyard supplies the routing decision while the user chooses the model pool. The surrounding service stays with the user's gateway or application. The project describes native OpenAI and Anthropic API compatibility, with flexible model selection and benchmarking across models and providers.

How Routing Works

The user configures the available models and picks a routing algorithm. The algorithm examines the request or the agent's recent tool activity, and some algorithms call a model to judge the task. The gateway or application then sends the request to the selected model, and routing can change as the agent keeps working.

• Auto: the current default preset, Execution (Stage) routing that prefers the efficient model, with a 0.5 confidence threshold and no classifier call.
• Task: a model judges whether the efficient model can handle the task.
• Execution: uses recent tool activity and outcome signals.
• Composite: combines Task and Execution, with a classifier setting the default tier when execution signals are uncertain.

The routing overview also covers Plan/Execute, escalation, advisor, sub-agent, and random strategies. Integration support varies by gateway.

Ways to Use It

Switchyard can be used through OpenRouter, a LiteLLM routing-plugin example (experimental, checkout-only), or a native NeMo Relay plugin. It can also run locally through a standalone proxy for demos and evaluation, or be embedded in a Rust application, with a Python embedding example. Guides exist for the pi and Oh My Pi agents.

Component Stability

The README labels switchyard-libsy as Beta, switchyard-llm-client and switchyard-runner as Alpha, and switchyard-server as Demo, not intended for production. It advises pinning the integrated version and evaluating the full agent, model pool, and routing configuration against a single-model baseline.