# llmbridge

> A sub-millisecond, drop-in OpenAI-compatible LLM gateway written in C++ that translates requests between AI providers with microsecond overhead.

llmbridge is an open-source C++ LLM gateway developed and maintained by Kottos AI, Inc. It sits between an application and a model provider, accepting OpenAI-compatible API requests and translating them to the upstream provider's dialect — Anthropic, Google Gemini, Cohere, AWS Bedrock, Azure OpenAI, and OpenAI-compatible services — then translating responses back. The project is licensed under Apache 2.0 and is currently in alpha, with the API described as unstable ahead of a planned v1.0 release.

## What It Is

llmbridge is a high-performance LLM proxy gateway built to HFT (High Frequency Trading) latency standards. Unlike Python-based gateways such as LiteLLM or Portkey, llmbridge is written in C++20 and uses a single-threaded `io_uring`/`epoll` event loop with no garbage collector pauses and no third-party runtime dependencies in its default build. Its core job is protocol translation: a client speaks the OpenAI API, and llmbridge rewrites each request to the upstream provider's wire format and rewrites the response back, adding microseconds rather than milliseconds of overhead. It can also be embedded as a C++ library for in-process translation without running a separate gateway process.

## Performance Architecture

The gateway is designed around a lean hot path with several concrete properties:

- **Zero-copy parsing:** `string_view` over the input buffer; output built in one growable buffer.
- **Hand-rolled, dependency-free JSON:** A recursive-descent parser scoped to chat-completion shapes, with no `nlohmann::json`, `jsoncpp`, or `simdjson` in the shipped binary.
- **No locks on the hot path:** Single-threaded `io_uring` event loop with a keep-alive upstream connection pool and no shared mutable state.
- **Hardened parsers:** Both parsers are continuously fuzzed under ASan/UBSan; the JSON parser is depth-limited and request bodies are size-capped.
- **Scale-out via `SO_REUSEPORT`:** One core sustains ~84k RPS non-streaming; additional workers are added with `--workers N`.

An independent benchmark by Jakub A. Wąsek of ENTERPILOT ran nine gateways against the same in-memory mock at 20,000 requests per variant. In that published measurement, llmbridge (v0.53.0) added 0.07 ms non-streaming and 0.18 ms streaming on a single core, while consuming 354 requests per percent of CPU — compared to GoModel at 23 and Bifrost at 11 in the same harness. Kottos AI's own head-to-head benchmark against LiteLLM showed llmbridge adding 80 µs p99 at 100 RPS versus LiteLLM's 87 ms on the same version tested, with llmbridge holding 41–80 µs p99 across 100–5,000 RPS.

## Provider and Feature Coverage

Current support for chat completions includes:

- **OpenAI ↔ Anthropic** — streaming (SSE, token-by-token) and non-streaming, including `stream_options.include_usage`
- **OpenAI ↔ Google Gemini** and **OpenAI ↔ Cohere** — non-streaming only
- **AWS Bedrock** (`--upstream-dialect bedrock`, v0.23.0) and **Azure OpenAI** (v0.24.0)
- **OpenAI-compatible providers** (Groq, Together, Fireworks, DeepInfra, Mistral) — passthrough, no body translation needed
- **Tool calling** — streaming and non-streaming, OpenAI ↔ Anthropic, including parallel calls and `tool_result` round-trips
- **Prompt caching** — `cache_control` breakpoints forwarded byte-for-byte on text parts, tools, and the system block
- **TLS** — opt-in build with OpenSSL ≥ 3.0; certificate and hostname verification always on

Not yet shipped and explicitly refused with an error message: vision/audio/file content parts, streaming for Gemini and Cohere, and Anthropic-in mode (an app written against the Anthropic SDK routing to an OpenAI-compatible upstream).

## Open-Core Model and Kottos AI Commercial Layer

The GitHub README describes llmbridge as "open-core." The repository contains the fast gateway core — translate and proxy to a single upstream. Multi-provider routing, the live provider price/latency book, observability, SSO, and a managed cloud are described as the commercial layer from Kottos AI. The hosted product records an "inference tape" — per-request metadata including tokens, cache usage, response timing, cost estimates, and errors — and uses that data to drive intelligent routing decisions across providers. A public trial dashboard is available without registration for Claude Code users via a two-line settings change.

## Current Status: Alpha (v0.53.0)

The project self-describes as alpha with an unstable API ahead of v1.0. The GitHub repository shows active development with the last push on 2026-09-16. The README states the v1.0 commitment will come after at least six months of public use and at least one production deployment with stable feedback. CI builds and tests run on Ubuntu 24.04 LTS against GCC 13, GCC 14, Clang 16, Clang 17, and Clang 18. The gateway is Linux-only by design (epoll, io_uring); the translator library is additionally built and tested on macOS. The project is founded by Lluís Antoni Jiménez Rugama, described as holding a PhD in applied mathematics with nine years in high-frequency and electronic trading.

## Features
- Drop-in OpenAI-compatible API gateway
- Sub-millisecond translation overhead (microseconds p99)
- OpenAI ↔ Anthropic translation, streaming and non-streaming
- OpenAI ↔ Google Gemini translation (non-streaming)
- OpenAI ↔ Cohere translation (non-streaming)
- AWS Bedrock and Azure OpenAI dialect support
- OpenAI-compatible provider passthrough (Groq, Together, Fireworks, etc.)
- Tool calling support streaming and non-streaming (OpenAI ↔ Anthropic)
- Prompt caching with cache_control breakpoint forwarding
- TLS upstream and inbound TLS termination (opt-in, OpenSSL ≥ 3.0)
- Per-request timing headers (x-llmbridge-*)
- io_uring event loop with epoll fallback
- Dependency-free default build
- Embeddable as a C++ library for in-process translation
- JSON configuration file with flag override
- Connection pooling with keep-alive upstream connections
- Hardened, continuously fuzzed parsers (ASan/UBSan)
- Scale-out via --workers N and SO_REUSEPORT
- Credential passthrough and scrubbing (never logged)
- Inference tape: per-request cost, cache, timing, and error recording

## Integrations
Anthropic Claude, Google Gemini, Cohere, AWS Bedrock, Azure OpenAI, OpenAI, Groq, Together AI, Fireworks AI, DeepInfra, Mistral, Claude Code, OpenSSL

## Platforms
WINDOWS, MACOS, LINUX, WEB, API, DEVELOPER_SDK, CLI

## Pricing
Open Source, Free tier available

## Version
v0.53.0

## Links
- Website: https://kottos.ai
- Documentation: https://github.com/kottos-ai/llmbridge#quick-start
- Repository: https://github.com/kottos-ai/llmbridge
- EveryDev.ai: https://www.everydev.ai/tools/llmbridge