# OmnisBench

> An open, reproducible benchmark for LLM routing efficiency that measures how close a routing policy gets to the ideal quality-per-dollar frontier using fresh, uncontaminated tasks.

OmnisBench is an Apache-2.0 open-source benchmark built by Fortitude Omnis Group to measure LLM routing efficiency — specifically, how close a given routing policy gets to the ideal quality-per-dollar frontier. It was created in response to a core problem with existing benchmarks: on old, widely-republished datasets like HumanEval and GSM8K, models have likely seen the answers during training, causing every policy to score near 100% and making routing appear pointless. OmnisBench addresses this by running evaluations on a fresh split of LiveCodeBench problems published after the models' training cutoffs.

## What It Is

OmnisBench is a CLI-driven evaluation framework for LLM routing policies. It scores multiple routing strategies — such as always calling the cheapest model, always calling the frontier model, random selection, or oracle (ideal per-request) routing — across both contaminated and fresh task splits. The benchmark publishes every model response in a `results.json` file, enabling anyone to re-derive the full leaderboard offline with a single `omnisbench verify` command and zero API calls. It is positioned as a complement to the academic RouterBench (arXiv:2403.12031) by staying live, cost-current, and continuously re-gradable.

## How the Fresh Split Changes the Picture

The contamination effect is the central finding OmnisBench documents. On the likely-contaminated split (HumanEval + GSM8K), all policies land near 100% quality, making routing appear to offer no benefit. On the fresh LiveCodeBench split (15 tasks from 2025 onward), the picture changes substantially:

- The cheapest model alone (gpt-5-nano) scores 60% on fresh tasks, down from 90% on contaminated ones
- Oracle (ideal) routing reaches 93.3% — above the frontier model's 86.7%
- Oracle routing costs approximately 60% less than always calling the frontier model
- The gap between contaminated and fresh scores is what OmnisBench calls "the prize routing was hiding"

## The Perturbation-Gap Probe

Beyond split labels and release dates, OmnisBench includes a `omnisbench probe` command that quantifies contamination rather than just tagging it. Each model is scored on original problems and on reworded copies that preserve the test cases but change the phrasing. A large drop in score when wording changes flags that the model relied on memorized phrasing rather than genuine reasoning. The probe runs with two independent rewriters (one from each provider family) to control for rewriter bias, and all results re-grade offline like the rest of the benchmark.

## Verifiability as a Design Principle

A stated design goal of OmnisBench is that every published number must be independently re-derivable. The `omnisbench verify` command re-runs graders against the stored responses in `results.json` and rebuilds the full leaderboard — quality and cost — without making any API calls. Altering a single stored answer causes verification to fail. The project explicitly contrasts this with vendor routing-savings claims that cannot be checked externally.

## Security Tradeoffs to Know

The README includes a prominent security warning: `omnisbench run` executes untrusted, model-generated Python code to grade code tasks. The v0 sandbox provides a fresh subprocess and a hard timeout but does **not** provide network isolation, filesystem isolation, or memory/resource limits. The project recommends running inside a container or disposable VM. Full sandbox hardening is listed as a tracked v1 roadmap item. The `omnisbench verify` command does not call live models but still re-runs graders against stored responses, so the same sandboxing caveat applies.

## Update: Run 2026-08-20 and Perturbation Probe

The headline run (2026-08-20) covers four models — Claude Opus 5, GPT-5, Claude Haiku 4.5, and GPT-5-nano — on 15 fresh LiveCodeBench tasks and 20 likely-contaminated tasks, with a pricing snapshot pinned to 2026-08-18. The perturbation probe runs were published on 2026-08-25 using both claude-sonnet-5 and gpt-5 as rewriters across 24 fresh problems. The repository was last pushed on 2026-09-16, and the roadmap lists widening the fresh task set and adding LiveCodeBench functional problems as next steps.

## Features
- Fresh-split LLM routing benchmark using LiveCodeBench 2025+ tasks
- Offline re-gradable leaderboard via omnisbench verify (zero API calls)
- Contamination detection via perturbation-gap probe
- Per-split leaderboards separating fresh and contaminated tasks
- Frontier escape rate reporting per routing policy
- Release-date margin tracking per task split
- Pinned pricing snapshots for reproducible cost comparisons
- Config-only addition of new datasets and models
- Oracle (ideal) routing ceiling as a benchmark target
- CLI commands: run, verify, report, probe

## Integrations
OpenAI (GPT-5, GPT-5-nano), Anthropic (Claude Opus 5, Claude Haiku 4.5, Claude Sonnet 5), LiveCodeBench (via Hugging Face datasets), HumanEval, GSM8K

## Platforms
CLI, API

## Pricing
Open Source

## Version
v0 (fresh-16k-2026-08-20)

## Links
- Website: https://omnisbench.fortitude-omnis.group/
- Documentation: https://github.com/Fortitude-Group/OmnisBench/blob/main/docs/perturbation-probe.md
- Repository: https://github.com/Fortitude-Group/OmnisBench
- EveryDev.ai: https://www.everydev.ai/tools/omnisbench