Helicone AI Gateway

Helicone AI Gateway is an open-source gateway from the Helicone team that lets applications call many LLM providers through one OpenAI-compatible API. It can be used as a cloud-hosted service or self-hosted via npx or Docker. The README lists it as Public Beta.

What It Is

The gateway sits between an application and LLM providers. Developers keep using the OpenAI SDK in any language, point the base URL at the gateway, and select models with provider-prefixed names such as openai/gpt-4o-mini. The gateway then handles provider selection, load balancing, caching, rate limiting, fallbacks, and tracing, and sends data to Helicone observability.

Routing, Caching, and Rate Limits

The README describes built-in routing strategies: model-based latency routing, provider latency-based P2C with PeakEWMA, weighted distribution, and cost optimization. Rate limits can be set per user, team, or globally by request count, token usage, or dollar amount. Response caching supports Redis and S3 backends, and the project says caching can cut costs and latency by up to 95%.

Deployment and Configuration

Self-hosted setups use provider API keys in a .env file and an optional config.yaml defining routers, load-balance strategies, cache stores, and rate limits, launched with npx @helicone/ai-gateway@latest. The cloud-hosted option offers a configuration wizard in the UI. Observability includes Helicone integration plus OpenTelemetry logs, metrics, and traces. The README publishes its own benchmark figures, such as P95 latency under 5ms, as vendor claims.