AgencyBench

AgencyBench is a benchmark from GAIR-NLP (SII) for evaluating LLM-based autonomous agents on long-horizon, real-world workflows. The V2 release contains 138 tasks across 32 scenarios and 6 agentic capabilities: Game, Frontend, Backend, Code, Research and MCP. It is released under the MIT license, with a project website, a paper on arXiv and a dataset on Hugging Face.

What It Is

AgencyBench is a set of scenario folders, each with a task specification (subtasks, deliverables, rubrics), a configuration file and an evaluation script. The README says scenarios require an average of about 90 tool calls, 1 million tokens and hours of execution. Examples include building a C++ chat app, a browser Sudoku game, a 3D solar system landing page, a deep-research report with citations, and a GitHub MCP workflow that creates an issue, branch and pull request.

How Automated Evaluation Works

A user simulation agent gives iterative feedback to the agent under test. Game and Frontend scenarios run in a Docker-based remote VM sandbox. The evaluator opens the generated deliverables in a browser, performs human-like actions such as mouse clicks, keyboard input and command-line operations, and records screenshots and video. Scoring is rubric-based and combines rule-based judges, vision-based judges and LLM-as-judge where appropriate. An optional visualize flag lets you watch the evaluation run.

Running a Scenario

Setup uses a Python 3.11 conda environment and the repository's requirements file. For Game and Frontend scenarios, start the agent-infra sandbox Docker image and point the scenario .env at it. Fill in the .env with the scaffold, the evaluated model API and the evaluator model configuration, then run eval_task.py inside a scenario folder. Each run writes a model-named directory with intermediate artifacts and a final meta_eval.json holding the evaluation details and score.

Leaderboard and Results

The project website hosts V2 and V1 leaderboards and task pages. The V2 leaderboard lists 9 evaluated agents with per-capability scores, an average score (Savg) and average attempts per task (Att). The authors report that closed-source models outperform open-source models (48.4% vs 32.1%). They also analyze resource efficiency, feedback-driven self-correction and tool-use preferences. The earlier V1 release has 10 tasks.