ARBR

ARBR is an open-source, MIT-licensed AI gateway and control plane built for teams running LLM workloads in production. It sits between applications and every major model provider, classifying and routing each request, enforcing budgets and guardrails, and logging structured cost and latency events — all before a single token reaches a provider. A hosted option is available at app.projectarbr.com for teams that prefer not to self-host.

What It Is

ARBR is a self-hosted model optimisation and governance control plane. It exposes a single OpenAI-compatible endpoint that any existing SDK, agent framework, or chat UI can point at without code changes. Behind that endpoint, ARBR classifies each request by task type and difficulty, applies human-configured routing rules or an AI routing policy, enforces per-application budgets and rate limits, and logs every call as a structured RequestRecord in MongoDB. The product is designed around a six-step optimisation lifecycle: observe real workloads, discover savings opportunities, build evaluation evidence, evaluate candidate models, approve and roll out changes, then verify the outcome with measured realised savings.

Architecture and Integration Model

ARBR can run standalone or in front of an existing LiteLLM proxy. Its gateway exposes two endpoints:

• Arbr-native (POST /v1/chat) — full attribution, task classification, routing rules, budgets, and caching
• OpenAI-compatible (POST /v1/chat/completions) — drop-in replacement for the OpenAI chat API, supporting SSE streaming

Provider connectivity spans Anthropic, OpenAI, Google Gemini, Amazon Bedrock, Azure OpenAI, Vertex AI, Groq, DeepSeek, Mistral, Moonshot, xAI, NVIDIA NIM, and any OpenAI-compatible or self-hosted endpoint. The model registry is MongoDB-backed and auto-syncs pricing and benchmark data from LiteLLM's public catalog, covering 3,000+ models according to the project homepage.

Human-Governed Routing by Design

A core design principle of ARBR is that no routing change happens without human approval. The cost guardrail and AI routing policy only activate after a human enables them. Recommendations — such as flagging premium-model overuse on cheap task types — remain advisory until accepted. Canary and shadow rollouts are built in, but promotion always requires a human decision. The project README states this is permanent by design, not a deferred feature.

Routing decisions are recorded on every call with a routingDecision field (explicit, passthrough, rule, auto, ai, cache, semantic_cache, fallback, budget, canary, external) and a routingExplain field capturing the non-derivable reasoning behind each decision.

Observability and Governance

Every request becomes a structured event attributed by application, workflow, team, model, provider, task type, and user. The dashboard surfaces:

• Total requests, cost, average latency, and realised savings over configurable windows
• Spend by provider, model, and application
• Cache hit rates and savings (with provider-specific cache token pricing)
• Per-user spend and realised savings (re-priced at the originally requested model's rate)

Events can be exported via OpenTelemetry to Datadog, Grafana, or Prometheus. Governance controls include per-application gateway API keys, budgets that alert, downgrade, or block at a configured cap, prompt-injection checks, output guardrails, and kill switches. Authentication supports a single admin key, OIDC (Okta, Auth0, Google Workspace, Keycloak), or trusted-header mode for per-user identity and role-based access.

Setup Path

ARBR ships with a demo mode that seeds realistic data so dashboards, the recommendation engine, and routing controls work without any provider keys. The one-command Docker path is:

git clone ... && docker compose up

The dashboard opens at http://localhost:4100. Provider keys can be added via the dashboard (encrypted at rest) or environment variables (which take precedence). JavaScript and Python client packages (arbr-client) are published to npm and PyPI. A standalone CLI (arbr-audit) audits a request log or wraps a live coding-agent session for cost analysis with no server or database required.

Update: v0.3.0

The latest release is v0.3.0, published July 22, 2026. The repository was created in June 2026 and last pushed in September 2026, indicating active early development. Features that the README notes have shipped since earlier versions include budgets, gateway API keys, governance controls, horizontal scale support via Docker Compose overlays, and OIDC authentication. A Helm chart and traffic-capture-to-fine-tuning-dataset feature are listed on the public roadmap as not yet shipped.