# Featherbench

> A single-file Python harness for building private LLM benchmarks from JSON tasks and comparing models on pass rate, cost and latency via OpenRouter.

Featherbench is an open-source, single-file Python harness for creating your own LLM benchmarks from real-world tasks. Tasks are small JSON files, and every selected model is run through one shared scaffold via OpenRouter, so pass rate, cost and latency can be compared directly. It is published under the MIT license and developed by Reinvently, according to the README.

## What It Is

Featherbench is a framework for teams that want a private task set that models have not seen, rather than relying on public leaderboards. The core harness, eval.py, uses two dependencies (openai as the OpenRouter client, and jinja2 for the HTML report). The README positions it as a lightweight, auditable alternative to heavier platforms such as Inspect AI, promptfoo and Braintrust.

## How a Benchmark Run Works

You write tasks as JSON files containing a prompt, an optional checker, optional tools and an optional rubric. Checkers include python_tests, regex, contains, not_contains, all, tool_called and tool_not_called. An optional LLM rubric, scored blind by one or more judge models, adds quality scoring on top of the pass/fail floor.

Models are defined in models.json with an OpenRouter slug, provider routing pins, sampling settings and reasoning effort. A single OPENROUTER_API_KEY covers all of them. Cost is taken from OpenRouter's per-response usage data.

## Outputs

Each run writes a timestamped JSONL file with per-trial records, a Markdown summary and a self-contained HTML review page. The summary reports pass rate with 95% Wilson confidence intervals, median latency, tokens, cost per model and a per-task grid. A judge-bias matrix is included for rubric tasks. The repo also ships a leaderboard file and a machine-readable summary.json with an Open Knowledge Format bundle.

## Safety and Setup

It requires Python 3.9+. Tasks using the python_tests checker execute model-generated code in a subprocess with a stripped environment and a timeout, but without filesystem or network isolation. The README recommends running on a sandboxed machine, for example with nono.

## Current Status

The latest listed release is V1.3, published 2026-08-22.

## Features
- Single-file Python harness
- Tasks defined as JSON files
- Deterministic checkers (python_tests, regex, contains, not_contains, tool_called)
- Optional cross-judged LLM rubric scoring
- Pass rate with 95% Wilson confidence intervals
- Cost and latency tracking per model
- JSONL, Markdown and HTML report outputs
- Routing-pinned model catalog via OpenRouter
- Hard refusal handling per task
- Tool-use checks for function-calling tasks
- Concurrent trial execution
- Unit test suite with mocked providers

## Integrations
OpenRouter, OpenAI Python client, Jinja2, nono

## Platforms
WINDOWS, MACOS, LINUX, API, CLI

## Pricing
Open Source

## Version
V1.3

## Links
- Website: https://github.com/ed-is-ai/featherbench
- Documentation: https://github.com/ed-is-ai/featherbench
- Repository: https://github.com/ed-is-ai/featherbench
- EveryDev.ai: https://www.everydev.ai/tools/featherbench