# AgencyBench

> Open-source benchmark that evaluates autonomous LLM agents on long-horizon, real-world tasks with automated rubric-based scoring.

AgencyBench is a benchmark from GAIR-NLP (SII) for evaluating LLM-based autonomous agents on long-horizon, real-world workflows. The V2 release contains 138 tasks across 32 scenarios and 6 agentic capabilities: Game, Frontend, Backend, Code, Research and MCP. It is released under the MIT license, with a project website, a paper on arXiv and a dataset on Hugging Face.

## What It Is

AgencyBench is a set of scenario folders, each with a task specification (subtasks, deliverables, rubrics), a configuration file and an evaluation script. The README says scenarios require an average of about 90 tool calls, 1 million tokens and hours of execution. Examples include building a C++ chat app, a browser Sudoku game, a 3D solar system landing page, a deep-research report with citations, and a GitHub MCP workflow that creates an issue, branch and pull request.

## How Automated Evaluation Works

A user simulation agent gives iterative feedback to the agent under test. Game and Frontend scenarios run in a Docker-based remote VM sandbox. The evaluator opens the generated deliverables in a browser, performs human-like actions such as mouse clicks, keyboard input and command-line operations, and records screenshots and video. Scoring is rubric-based and combines rule-based judges, vision-based judges and LLM-as-judge where appropriate. An optional visualize flag lets you watch the evaluation run.

## Running a Scenario

Setup uses a Python 3.11 conda environment and the repository's requirements file. For Game and Frontend scenarios, start the agent-infra sandbox Docker image and point the scenario .env at it. Fill in the .env with the scaffold, the evaluated model API and the evaluator model configuration, then run eval_task.py inside a scenario folder. Each run writes a model-named directory with intermediate artifacts and a final meta_eval.json holding the evaluation details and score.

## Leaderboard and Results

The project website hosts V2 and V1 leaderboards and task pages. The V2 leaderboard lists 9 evaluated agents with per-capability scores, an average score (Savg) and average attempts per task (Att). The authors report that closed-source models outperform open-source models (48.4% vs 32.1%). They also analyze resource efficiency, feedback-driven self-correction and tool-use preferences. The earlier V1 release has 10 tasks.

## Features
- 138 tasks across 32 real-world long-horizon scenarios
- Six capability areas: Game, Frontend, Backend, Code, Research, MCP
- User simulation agent for iterative feedback
- Docker VM sandbox for visual and functional evaluation
- Rubric-based scoring with rule-based, vision-based and LLM judges
- Optional visualization of the evaluation process
- Per-run meta_eval.json score output
- Public V1 and V2 leaderboards

## Integrations
Docker, Hugging Face, GitHub MCP, Conda

## Platforms
WEB, API, CLI

## Pricing
Open Source

## Version
V2

## Links
- Website: https://agencybench.opensii.ai
- Documentation: https://github.com/GAIR-NLP/AgencyBench#readme
- Repository: https://github.com/GAIR-NLP/AgencyBench
- EveryDev.ai: https://www.everydev.ai/tools/agencybench