# LiteLLM

> Open source AI Gateway and Python SDK to call 100+ LLMs in OpenAI format, with cost tracking, guardrails, load balancing, and virtual key management.

LiteLLM is an open source AI Gateway and Python SDK built by BerriAI, backed by Y Combinator (W23), that gives platform and ML teams a single unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and many more — all using the OpenAI format. It is available as a self-hosted proxy server or as a Python library, and the GitHub repository has accumulated over 47,000 stars and more than 1,000 contributors as of mid-2026.

## What It Is

LiteLLM sits between your application code and the fragmented landscape of LLM provider APIs. Instead of maintaining separate SDKs, authentication patterns, and request formats for every model provider, developers and platform teams route all LLM calls through LiteLLM's unified interface. The project ships in two complementary forms: a **Python SDK** for direct library integration in application code, and an **AI Gateway (Proxy Server)** that acts as a centralized service for entire teams or organizations.

## Core Capabilities

The open source tier covers a broad set of production-ready features:

- **100+ LLM provider integrations** — OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Cohere, Mistral, Groq, Ollama, HuggingFace, and dozens more, across endpoints including `/chat/completions`, `/embeddings`, `/images`, `/audio`, `/batches`, `/rerank`, and the A2A agent protocol.
- **Drop-in OpenAI compatibility** — swap providers without rewriting application code; the proxy exposes an OpenAI-compatible API surface.
- **Virtual keys** — issue scoped API keys to teams and users without exposing underlying provider credentials.
- **Spend tracking and budgets** — attribute cost to individual keys, users, teams, or organizations; supports tag-based tracking and logging to S3, GCS, and other destinations.
- **Load balancing and fallbacks** — route across multiple deployments with configurable retry and fallback logic; the README cites 8ms P95 latency at 1,000 RPS in published benchmarks.
- **LLM guardrails** — set content and safety guardrails per request or per key/team.
- **Observability** — integrates with Langfuse, Langsmith, Arize Phoenix, OpenTelemetry, Datadog, Prometheus, and more.
- **MCP Gateway** — connect MCP servers to any LLM through the proxy, with Cursor IDE support documented.
- **A2A Agent Gateway** — invoke agents built on LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, and Pydantic AI via the A2A protocol.

## Deployment Model

LiteLLM is designed to be self-hosted. The proxy server ships as a Docker image (the homepage states 240M+ Docker pulls), and the README documents quick-start paths via Docker Compose, Render, and Railway. A `-stable` Docker tag undergoes 12-hour load tests before publication. An enterprise-hosted (SaaS) option is also available for organizations that prefer not to manage infrastructure themselves.

## Enterprise Tier

The enterprise offering adds features on top of the open source base, available in both cloud and self-managed deployment:

- SSO and SCIM provisioning
- OIDC/JWT authentication with group-based access for automatic token generation
- Audit logs and secret manager integrations
- Virtual key rotation
- Team and org admin delegation
- Pagerduty alerting and advanced Prometheus metrics
- 24/7 support with custom SLAs and live upgrade assistance

The enterprise page lists a 30-day trial key available via a sales interest form, and pricing is available on request.

## Update: v1.85.0

The latest release as of mid-May 2026 is **v1.85.0**, published on May 17, 2026. The repository is actively maintained on the `litellm_internal_staging` branch, with the last push recorded on May 20, 2026. The project's GitHub topics include `mcp-gateway` and `a2a`, reflecting recent additions of MCP and agent-to-agent protocol support as product-direction signals. The changelog is published at docs.litellm.ai/release_notes, and a town hall to discuss the roadmap was announced for May 19, 2026.

## Features
- Unified API for 100+ LLM providers
- OpenAI-compatible proxy server
- Python SDK for direct integration
- Virtual keys for access control
- Spend tracking by key/user/team/org
- Budget and rate limit enforcement
- LLM fallbacks and load balancing
- LLM guardrails per request or key/team
- Observability integrations (Langfuse, Langsmith, Arize Phoenix, OTEL, Datadog)
- Prometheus metrics
- S3/GCS/Azure Data Lake logging
- MCP Gateway for connecting MCP servers to any LLM
- A2A Agent Gateway
- SSO and SCIM (Enterprise)
- OIDC/JWT authentication (Enterprise)
- Audit logs (Enterprise)
- Secret manager integrations (Enterprise)
- Virtual key rotation (Enterprise)
- Admin dashboard UI
- Tag-based spend tracking
- Pass-through endpoints
- Batches API support
- Embeddings, image generation, audio transcription endpoints

## Integrations
OpenAI, Anthropic, Azure OpenAI, Google Gemini, Google Vertex AI, AWS Bedrock, AWS Sagemaker, Cohere, Mistral AI, Groq, Ollama, HuggingFace, Langfuse, Langsmith, Arize Phoenix, OpenTelemetry, Datadog, Prometheus, S3, GCS, Azure Data Lake, PagerDuty, Slack, Discord, Microsoft Teams, Webhook, LangGraph, Pydantic AI, Cursor IDE, Fireworks AI, Together AI, Perplexity AI, DeepSeek, IBM Watsonx, Databricks, Snowflake, ElevenLabs, AssemblyAI, Deepgram, Replicate, OpenRouter, VLLM, Nvidia NIM

## Platforms
API, CLI, DEVELOPER_SDK, WEB

## Pricing
Open Source, Free tier available

## Links
- Website: https://www.litellm.ai
- Documentation: https://docs.litellm.ai/docs/
- Repository: https://github.com/BerriAI/litellm
- EveryDev.ai: https://www.everydev.ai/tools/litellm