Supabase Evals

Supabase Evals is an open-source benchmarking framework published by Supabase under the Apache License 2.0. It measures how effectively AI coding agents — such as Claude Code, OpenAI Codex, and OpenCode — can complete real Supabase developer tasks in realistic, stateful environments. The project lives at github.com/supabase/evals and its results are published publicly at supabase.com/evals.

What It Is

Supabase Evals is an agent evaluation harness purpose-built for the Supabase platform. Rather than testing generic coding ability, it evaluates agents across the specific stages of the Supabase developer journey: Build, Deploy, Investigate, and Resolve. Each eval scenario provides agents with a realistic project environment — including database state, observability logs, and deployed edge functions — and scores their output using SQL checks, real client calls, file inspection, and optionally an LLM judge for broader assessment.

How the Eval Framework Works

The framework is organized around three core concepts:

• Evals — individual scenarios under evals/<id>/, each containing a PROMPT.md task description, an EVAL.ts scorer, and optional starting state split between a remote/ directory (the hosted project's database, logs, and functions) and a local/ directory (the agent's working files).
• Experiments — named agent/runtime/model configurations defined in experiments/<name>.ts, such as claude-code-sonnet-5 or codex-gpt5.
• Runtimes — two modes are supported: a tools eval mode where the agent operates against an MCP/tool surface backed by platform-lite (a lightweight Supabase Management API-compatible layer), and a local-stack eval mode where the agent runs inside a Docker sandbox with the real Supabase CLI installed.

Scoring draws on SQL checks against the project state, real client calls made as simulated users, and inspection of files the agent creates. When broader judgment is needed, an LLM judge reviews the result.

Skills and Progressive Disclosure

Experiments can be configured with skills sourced from the supabase/agent-skills submodule. Skills are loaded lazily using progressive disclosure: only each skill's name and description appear in the system prompt, and the agent pulls full instructions on demand. In local-stack mode, skills are installed into the workspace via Vercel's skills CLI; in tools mode, a load_skill tool returns the full instructions when called. This design lets experiments test agents both with and without domain-specific Supabase knowledge, enabling head-to-head comparisons labeled "With skills" vs. "Without skills" on the public leaderboard.

Architecture and Deployment Model

The framework uses pnpm as its package manager and TypeScript as its primary language. Key packages include @supabase-evals/sandbox for Docker-based local-stack orchestration and @supabase/mcp-server-supabase for MCP tool surfaces. The platform-lite runtime exposes a Supabase Management API-compatible HTTP surface backed by @supabase/lite, allowing real MCP tools to run against a lightweight project without a full hosted environment. Results are written to local JSON files and can be exported for viewing in a bundled web app (pnpm web).

Local-stack evals require a running Docker daemon. Each attempt boots a fresh sandbox container that mounts the host Docker socket, so supabase start spawns the local stack as sibling containers with host networking. Supabase's default host ports (54321–54329) must be free before running.

Current Status and Leaderboard

The repository was created in April 2026 and was last pushed in July 2026, indicating active development. The public leaderboard at supabase.com/evals shows benchmark results across agents including Codex/GPT-5.6 sol, Claude Code/Opus 5, Claude Code/Sonnet 5, OpenCode/Kimi K3, and Codex/GPT-5.4 mini, scored across Build, Deploy, Investigate, and Resolve dimensions. The leaderboard is filterable by agent, journey stage, and product area. The project accepts community contributions via CONTRIBUTING.md and tracks open issues on GitHub.