# Supabase Evals

> An open-source evaluation framework that benchmarks how well AI agents perform across the Supabase developer journey, from building and deploying to investigating and resolving production issues.

Supabase Evals is an open-source benchmarking framework published by Supabase under the Apache License 2.0. It measures how effectively AI coding agents — such as Claude Code, OpenAI Codex, and OpenCode — can complete real Supabase developer tasks in realistic, stateful environments. The project lives at `github.com/supabase/evals` and its results are published publicly at `supabase.com/evals`.

## What It Is

Supabase Evals is an agent evaluation harness purpose-built for the Supabase platform. Rather than testing generic coding ability, it evaluates agents across the specific stages of the Supabase developer journey: **Build**, **Deploy**, **Investigate**, and **Resolve**. Each eval scenario provides agents with a realistic project environment — including database state, observability logs, and deployed edge functions — and scores their output using SQL checks, real client calls, file inspection, and optionally an LLM judge for broader assessment.

## How the Eval Framework Works

The framework is organized around three core concepts:

- **Evals** — individual scenarios under `evals/<id>/`, each containing a `PROMPT.md` task description, an `EVAL.ts` scorer, and optional starting state split between a `remote/` directory (the hosted project's database, logs, and functions) and a `local/` directory (the agent's working files).
- **Experiments** — named agent/runtime/model configurations defined in `experiments/<name>.ts`, such as `claude-code-sonnet-5` or `codex-gpt5`.
- **Runtimes** — two modes are supported: a **tools eval** mode where the agent operates against an MCP/tool surface backed by `platform-lite` (a lightweight Supabase Management API-compatible layer), and a **local-stack eval** mode where the agent runs inside a Docker sandbox with the real Supabase CLI installed.

Scoring draws on SQL checks against the project state, real client calls made as simulated users, and inspection of files the agent creates. When broader judgment is needed, an LLM judge reviews the result.

## Skills and Progressive Disclosure

Experiments can be configured with **skills** sourced from the `supabase/agent-skills` submodule. Skills are loaded lazily using progressive disclosure: only each skill's name and description appear in the system prompt, and the agent pulls full instructions on demand. In local-stack mode, skills are installed into the workspace via Vercel's `skills` CLI; in tools mode, a `load_skill` tool returns the full instructions when called. This design lets experiments test agents both with and without domain-specific Supabase knowledge, enabling head-to-head comparisons labeled "With skills" vs. "Without skills" on the public leaderboard.

## Architecture and Deployment Model

The framework uses `pnpm` as its package manager and TypeScript as its primary language. Key packages include `@supabase-evals/sandbox` for Docker-based local-stack orchestration and `@supabase/mcp-server-supabase` for MCP tool surfaces. The `platform-lite` runtime exposes a Supabase Management API-compatible HTTP surface backed by `@supabase/lite`, allowing real MCP tools to run against a lightweight project without a full hosted environment. Results are written to local JSON files and can be exported for viewing in a bundled web app (`pnpm web`).

Local-stack evals require a running Docker daemon. Each attempt boots a fresh sandbox container that mounts the host Docker socket, so `supabase start` spawns the local stack as sibling containers with host networking. Supabase's default host ports (54321–54329) must be free before running.

## Current Status and Leaderboard

The repository was created in April 2026 and was last pushed in July 2026, indicating active development. The public leaderboard at `supabase.com/evals` shows benchmark results across agents including Codex/GPT-5.6 sol, Claude Code/Opus 5, Claude Code/Sonnet 5, OpenCode/Kimi K3, and Codex/GPT-5.4 mini, scored across Build, Deploy, Investigate, and Resolve dimensions. The leaderboard is filterable by agent, journey stage, and product area. The project accepts community contributions via `CONTRIBUTING.md` and tracks open issues on GitHub.

## Features
- Agent benchmarking across Build, Deploy, Investigate, and Resolve stages
- Realistic stateful environments with database, logs, and edge functions
- Two eval runtimes: tools mode (MCP/platform-lite) and local-stack mode (Docker + Supabase CLI)
- SQL-based scoring, real client call verification, and LLM judge support
- Progressive skill disclosure via supabase/agent-skills submodule
- Head-to-head experiment comparisons with and without skills
- Public leaderboard at supabase.com/evals
- Exportable results with bundled web app viewer
- Support for multiple agents: Claude Code, Codex, OpenCode
- TypeScript-based eval and experiment definitions

## Integrations
Claude Code, OpenAI Codex, OpenCode, Supabase CLI, Supabase MCP Server, platform-lite, Docker, Vercel Skills CLI, pnpm

## Platforms
CLI, WEB, API

## Pricing
Open Source

## Version
main

## Links
- Website: https://supabase.com/evals
- Documentation: https://github.com/supabase/evals/blob/main/CONTRIBUTING.md
- Repository: https://github.com/supabase/evals
- EveryDev.ai: https://www.everydev.ai/tools/supabase-evals