# Preseason

> An open-source benchmark that measures which developer tools LLMs recommend when asked to build real web apps, publishing rankings and head-to-head comparisons.

Preseason is an open-source benchmark built by Beto Muniz (betocmn) that tracks which developer tools AI language models recommend when prompted to build real web applications. It runs a frozen set of vibe-coding prompts across a fixed panel of models, parses each response for tool recommendations, and publishes the results as rankings, head-to-head matches, and methodology notes at preseason.ai. The project is licensed under MIT and welcomes community contributions.

## What It Is

Preseason sits at the intersection of LLM evaluation and developer-tool discovery. Rather than asking "which tool is objectively better," it asks a more tractable question: which tools do today's AI coding assistants actually recommend when a developer asks them to build something? The benchmark runs prompts at multiple skill levels — from beginner to expert engineer — against a rotating panel of models via OpenRouter, then parses each model's structured appendix into a case decision: `tool`, `none`, or `invalid`. Results are aggregated into category rankings (databases, auth, hosting, payments, email, background jobs, UI libraries, observability, AI/model providers, and more) and published on the live site.

## How the Benchmark Works

The system uses a cron route (`/api/cron/benchmark-run`) that fires every six minutes, walking every prompt × model combination in the active season. Each season freezes a specific set of prompt versions and model snapshots, making results reproducible and inspectable directly in the GitHub repository. The model is required to produce a strict machine-readable appendix; unrecognized tool names are held in a candidate queue for admin review rather than guessed at. Only runs that pass QC are published to the public-facing rankings and match pages.

```
Cron → OpenRouter (one model) → Response parser → Case decision → Rankings + matches
```

## Tech Stack and Architecture

Preseason is built with a modern TypeScript stack:

- **Next.js 15** (App Router, React Server Components) for the frontend and API routes
- **tRPC v11** for typed end-to-end API calls
- **Drizzle ORM** + **Supabase** (Postgres + email-OTP auth) for data persistence
- **OpenRouter** as the model gateway, enabling multi-model benchmarking
- **Tailwind CSS v4** + **shadcn/ui** for the UI
- **Vitest** + Testcontainers for integration-tested Postgres
- **Biome** for linting and formatting

The supported deployment path is Vercel + Supabase Cloud, with a one-click Vercel deploy button available in the repository.

## Why It Matters for the Vibe-Coding Era

The README notes that recommendations from AI coding assistants shape developer tool adoption faster than blog posts or social media threads. If a foundation model quietly favors one database or hosting provider, that preference scales to every developer using it. Preseason makes the methodology behind those recommendations open, reproducible, and contestable — anyone can see what today's LLMs recommend, run their own benchmark on custom prompts or a different model panel, and file issues when results look off with a public paper trail.

## Current Status and Limitations

The project was created in February 2026 and last updated in August 2026, with active CI and CodeQL workflows. The README explicitly notes current limitations: the benchmark measures recommendations, not objective tool quality; results depend on the frozen prompt set and model snapshots; and the project is early, so rankings should be treated as directional rather than definitive. Self-hosting via Docker Compose or plain Postgres is not yet supported because the project currently depends on Supabase Auth.

## Features
- Frozen prompt benchmarks across skill levels (beginner to expert)
- Fixed panel of LLMs via OpenRouter model gateway
- Category rankings for databases, auth, hosting, payments, email, background jobs, UI libraries, observability, and more
- Head-to-head tool comparisons
- Strict machine-readable response parsing with candidate queue for unknown tools
- Season-based freezing of prompt versions and model snapshots
- Public methodology and reproducible results
- Open-source MIT-licensed codebase
- One-click Vercel deploy
- Integration-tested Postgres with Vitest + Testcontainers

## Integrations
OpenRouter, Supabase, Vercel, Next.js, tRPC, Drizzle ORM, Tailwind CSS, shadcn/ui, Biome, Vitest, GitHub Actions

## Platforms
WEB, API, CLI

## Pricing
Open Source

## Version
main

## Links
- Website: https://preseason.ai
- Documentation: https://github.com/betocmn/preseason/blob/main/docs/SETUP.md
- Repository: https://github.com/betocmn/preseason
- EveryDev.ai: https://www.everydev.ai/tools/preseason