harness-evals

harness-evals is an open-source evaluation repository run by Garry Tan that compares AI agent harnesses (the system prompt, tools and memory wrapped around a model) while holding the model and tasks constant. It measures how often each harness solves tasks, what each solved task costs, and how long each try takes.

What It Is

The repository publishes benchmark results for harnesses such as Claude Code, Codex, OpenClaw, Hermes Agent, Pi and QM. Tests run on Claude Opus 5.5 and GPT-6.1 Sol, using 52 public Terminal-Bench 2.1 coding tasks and 20 multi-user tasks in which the agent acts as a simulated company's chat bot. Rankings are by cost per solved task, with 95% ranges, pass rates and median time per try.

What the Results Cover

The README reports that harnesses differed more on cost than on pass rate, and recommends comparing on cost per solved task using your own work. Separate findings examine where must-follow rules are best placed (system prompt or per-turn hook), the cost breakdown of different harnesses, prompt cuts, and shared-brain access rules for gbrain. It also tests gstack skills, reporting that Opus 5.5 read them but did not follow their steps. The README notes these are early results on public tasks and pinned harness versions.

Running and Verifying It

The keyless steps call no model and need no API key, only Bun and git. Users clone the repository, install dependencies, rebuild the scoreboard from committed results and check it matches, and recompute the multi-user tables from saved tries. Docs describe replicating a finding with your own key under a spending cap. The README includes disclosures about Garry Tan's connections to several of the tested projects.