# VernLLM

> An open-source TypeScript framework for resilient LLM calls with retry budgets, circuit breaking, provider fallback, rate limiting, caching, and middleware — all running in-process with zero runtime dependencies.

VernLLM is an open-source TypeScript library that wraps your existing LLM provider clients with a production-grade resilience layer. It runs entirely in your own process — no extra network hop, no proxy — and ships with zero runtime dependencies. The latest release is vern-llm@2.8.0, published in September 2026.

## What It Is

VernLLM is an LLM call framework that sits between your application code and provider SDKs (OpenAI, Anthropic, Gemini, AWS Bedrock, and dozens of OpenAI-compatible endpoints). Rather than replacing your provider client, it wraps it through a small adapter interface and adds configurable resilience primitives: retries with backoff and jitter, a circuit breaker, multi-provider fallback, local rate limiting, response caching, and request/response middleware. The core export is a single `VernLLM` class that accepts all options at construction time and exposes a typed `call` / `cachedCall` API.

## Core Resilience Primitives

Each primitive is independently configurable and can be replaced with a custom implementation:

- **Retry budget** — caps what fraction of recent traffic can be retries within a rolling time window, preventing retry storms
- **Circuit breaker** — stops cascading failures by halting calls to a provider that is repeatedly failing
- **Provider fallback** — automatically routes to a backup provider (e.g., Anthropic when OpenAI is down) within the same process
- **Rate limiting** — queues requests locally under per-minute request, token, and concurrency ceilings
- **Timeout** — prevents hanging calls from blocking indefinitely
- **Caching** — `cachedCall` accepts any adapter implementing `get`/`set`, so Redis, a database, or a custom store can replace the built-in in-memory cache
- **Middleware** — `transform` edits or redacts outgoing requests; `wrap` runs around the full logical call including retries and fallback attempts, enabling logging, tracing, or cost tracking

## Provider Coverage

VernLLM ships first-class adapters for OpenAI (`fromOpenAI`), Anthropic (`fromAnthropic`), Google Gemini (`fromGemini`), and AWS Bedrock. Any OpenAI-compatible endpoint — including Groq, Mistral, DeepSeek, Cerebras, Together AI, Fireworks AI, Ollama, OpenRouter, Perplexity, DeepInfra, LM Studio, vLLM, xAI (Grok), NVIDIA NIM, Vercel AI Gateway, Cloudflare Workers AI, SambaNova, Lambda Labs, Snowflake Cortex, and more — is supported through a shared `fromOpenAI`-compatible path. A `fromFetch` adapter covers any provider reachable over HTTP.

## Architecture and Tradeoffs

VernLLM is deliberately in-process rather than a gateway. The homepage explains the tradeoff directly: a gateway is the better choice for one shared setup across many services or languages, while VernLLM is the better choice when you want to react to your own application logic, avoid an extra network hop, or keep the resilience layer close to the call site. The library is written in TypeScript from the ground up; structured output schemas, call parameters, and errors are all typed so mistakes surface at compile time. It does not bundle Zod or any provider SDK — you bring your own clients and schema validators.

## Update: vern-llm@2.8.0

The repository was created in July 2026 and has been actively developed, with the latest release (vern-llm@2.8.0) published on September 8, 2026. The project is a pnpm monorepo with two workspaces: the `packages/vern-llm` npm package and an `apps/docs` Fumadocs-powered documentation site at vernllm.dev. GitHub CI workflows cover build checks, linting, type checking, unit tests, integration tests, CodeQL analysis, and code coverage via Codecov. The project carries an OpenSSF Best Practices badge and is licensed MIT.

## Features
- Retry budgets with configurable rolling windows
- Circuit breaker to stop cascading failures
- Multi-provider fallback (e.g., OpenAI → Anthropic)
- Local in-process rate limiting (requests/min, tokens/min, concurrency)
- Configurable timeout per call
- Response caching via cachedCall with pluggable cache adapters
- Request transform middleware for editing or redacting outgoing requests
- Wrap middleware for logging, tracing, and cost tracking
- Zero runtime dependencies
- Full TypeScript types for params, structured outputs, and errors
- Adapters for OpenAI, Anthropic, Gemini, AWS Bedrock, and OpenAI-compatible providers
- fromFetch adapter for any HTTP-reachable provider
- Structured output schema support
- Default reasoning effort and max tokens configuration
- Backoff and jitter on retries

## Integrations
OpenAI, Anthropic, Google Gemini, AWS Bedrock, Groq, Mistral, DeepSeek, Cerebras, Together AI, Fireworks AI, Ollama, OpenRouter, Perplexity, DeepInfra, LM Studio, vLLM, xAI (Grok), NVIDIA NIM, Vercel AI Gateway, Cloudflare Workers AI, SambaNova Cloud, Lambda Labs, Snowflake Cortex, Anyscale, Lepton AI, Inference.net, Infermatic, Novita AI, Hyperbolic, Moonshot (Kimi), Zhipu (GLM), Baseten, Featherless AI, Friendli AI, SiliconFlow, Parasail, StepFun, MiniMax, Nebius AI Studio, Redis (via custom cache adapter)

## Platforms
API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
vern-llm@2.8.0

## Links
- Website: https://vernllm.dev
- Documentation: https://vernllm.dev/docs
- Repository: https://github.com/LakBud/vernLLM
- EveryDev.ai: https://www.everydev.ai/tools/vernllm