Preseason

Preseason is an open-source benchmark built by Beto Muniz (betocmn) that tracks which developer tools AI language models recommend when prompted to build real web applications. It runs a frozen set of vibe-coding prompts across a fixed panel of models, parses each response for tool recommendations, and publishes the results as rankings, head-to-head matches, and methodology notes at preseason.ai. The project is licensed under MIT and welcomes community contributions.

What It Is

Preseason sits at the intersection of LLM evaluation and developer-tool discovery. Rather than asking "which tool is objectively better," it asks a more tractable question: which tools do today's AI coding assistants actually recommend when a developer asks them to build something? The benchmark runs prompts at multiple skill levels — from beginner to expert engineer — against a rotating panel of models via OpenRouter, then parses each model's structured appendix into a case decision: tool, none, or invalid. Results are aggregated into category rankings (databases, auth, hosting, payments, email, background jobs, UI libraries, observability, AI/model providers, and more) and published on the live site.

How the Benchmark Works

The system uses a cron route (/api/cron/benchmark-run) that fires every six minutes, walking every prompt × model combination in the active season. Each season freezes a specific set of prompt versions and model snapshots, making results reproducible and inspectable directly in the GitHub repository. The model is required to produce a strict machine-readable appendix; unrecognized tool names are held in a candidate queue for admin review rather than guessed at. Only runs that pass QC are published to the public-facing rankings and match pages.

Cron → OpenRouter (one model) → Response parser → Case decision → Rankings + matches

Tech Stack and Architecture

Preseason is built with a modern TypeScript stack:

• Next.js 15 (App Router, React Server Components) for the frontend and API routes
• tRPC v11 for typed end-to-end API calls
• Drizzle ORM + Supabase (Postgres + email-OTP auth) for data persistence
• OpenRouter as the model gateway, enabling multi-model benchmarking
• Tailwind CSS v4 + shadcn/ui for the UI
• Vitest + Testcontainers for integration-tested Postgres
• Biome for linting and formatting

The supported deployment path is Vercel + Supabase Cloud, with a one-click Vercel deploy button available in the repository.

Why It Matters for the Vibe-Coding Era

The README notes that recommendations from AI coding assistants shape developer tool adoption faster than blog posts or social media threads. If a foundation model quietly favors one database or hosting provider, that preference scales to every developer using it. Preseason makes the methodology behind those recommendations open, reproducible, and contestable — anyone can see what today's LLMs recommend, run their own benchmark on custom prompts or a different model panel, and file issues when results look off with a public paper trail.

Current Status and Limitations

The project was created in February 2026 and last updated in August 2026, with active CI and CodeQL workflows. The README explicitly notes current limitations: the benchmark measures recommendations, not objective tool quality; results depend on the frozen prompt set and model snapshots; and the project is early, so rankings should be treated as directional rather than definitive. Self-hosting via Docker Compose or plain Postgres is not yet supported because the project currently depends on Supabase Auth.