Claw-Eval

Claw-Eval is an open-source evaluation harness and benchmark for LLM-based autonomous agents, published by the claw-eval organization under the MIT License. It provides 300 human-verified tasks with 2,159 rubrics across 9 categories, and a public leaderboard at claw-eval.github.io. Agents are graded on completion, safety and robustness.

What It Is

Claw-Eval is a Python benchmark harness that runs models as agents against tasks and audits their full trajectories. The README describes the title as "Towards Trustworthy Evaluation of Autonomous Agents" and links an arXiv paper and datasets on Hugging Face and ModelScope. Tasks are split into general (161), multimodal (101) and multi-turn (38) sets. General tasks cover communication, finance, ops and productivity. Multimodal tasks cover webpage generation, video QA and document extraction. Multi-turn tasks use simulated user personas.

How Scoring Works

The primary metric is Pass^3: a task counts as passed only if the model meets the success criteria in all three independent trials. The site gives the per-task score as safety × (0.8 × completion + 0.2 × robustness), with a pass threshold of 75. The leaderboard reports Pass^3, Pass@3, completion, robustness, safety and average score, plus a view of Pass^3 against token efficiency on general tasks. According to the README, graders are gemini-3-flash for general and multimodal tasks and claude opus4.6 for multi-turn tasks.

Running It

Setup uses uv with Python 3.11. Users set an OPENROUTER_API_KEY, optionally a SERP_DEV_KEY for tasks needing real web search, and run scripts/test_sandbox.sh. Evaluations are launched with the claw-eval batch command using a model config YAML, with options such as sandbox mode, number of trials and parallelism. Full fixtures, including videos, are hosted on Hugging Face. EvalScope also supports running Claw-Eval against OpenAI-compatible endpoints.

Current Status

The README lists v1.1.0 with 300 human-verified tasks in 9 categories. It states that the codebase is being audited so leaderboard results can be community-verified. Teams wanting to submit results to the leaderboard are directed to the listed contact emails.