claw-eval
Claw-Eval is an open-source benchmark and evaluation harness for trustworthy evaluation of autonomous LLM agents. It evaluates realistic personal-assistant workflows with tool use, sandbox files, multimodal inputs and simulated user interactions, using trajectory-aware grading of completion, safety and robustness.
At a Glance
- AI research teams and universities
- Foundation-model developers
- Autonomous-agent and personal-assistant developers
- AI safety, evaluation and reliability researchers
- +1 more
AI Tools by claw-eval
(1)
Claw-Eval
Open Source LLM Agent Benchmark
Discussions
No discussions yet
Be the first to start a discussion about claw-eval
Latest News
README updated with project usage-statistics badges and current benchmark information
Repository added broader feature support and a Gemini-native path for the LLM judge
EvalScope integration note added for running Claw-Eval with standardized evaluation workflows
Claw-Eval paper revised to arXiv version 3
Products & Services
Open-source Python harness for evaluating LLM-as-agent systems through 300 human-verified tasks across general, multimodal and multi-turn splits, with sandbox execution, full-trajectory auditing and standardized scoring.
Dataset distributed through Hugging Face and ModelScope, containing task queries, fixtures, language metadata and categories for the benchmark.
A separate live-evaluation companion linked from the official Claw-Eval site.
Market Position
Claw-Eval positions itself as an auditable, end-to-end alternative to narrower or less transparent agent benchmarks. Its paper contrasts it with AgentBench, GAIA, WebArena, VisualWebArena, OSWorld, Terminal-Bench and PinchBench, emphasizing its combination of multimodal and multi-turn coverage, explicit safety and robustness scoring, perturbation/error-injection analysis, sandboxing and trajectory evidence.
Leadership
Founders
Bowen Ye
Core contributor affiliated with Peking University; GitHub maintainer pkuYmiracle.
Rang Li
Core contributor affiliated with Peking University; corresponding/submission contact for the Claw-Eval paper.
Qibin Yang
Core contributor affiliated with Peking University.
Zhihui Xie
Core contributor affiliated with the University of Hong Kong.
Yuanxin Liu
Core contributor affiliated with Peking University.
Linli Yao
Core contributor affiliated with Peking University.
Hanglong Lyu
Core contributor affiliated with Peking University.
Lei Li
Core contributor and project lead affiliated with the University of Hong Kong.
Executive Team
Lei Li
Project lead
Core contributor affiliated with the University of Hong Kong.
Board of Directors
Founding Story
The project was started to address shortcomings in existing agent benchmarks: trajectory-opaque grading, underspecified safety and robustness evaluation, and narrow coverage of modalities and interaction paradigms. Its initial vision was an end-to-end, reproducible evaluation suite grounded in realistic workflows, human-verified tasks, sandboxed execution and auditable evidence.
Business Model
Revenue Model
The project is distributed as MIT-licensed open-source software and public datasets; the official materials describe research, community contributions and integrations rather than a paid commercial offering.
Target Markets
- AI research teams and universities
- Foundation-model developers
- Autonomous-agent and personal-assistant developers
- AI safety, evaluation and reliability researchers
- Organizations assessing agents for real-world deployment
- Benchmarking autonomous LLM agents on realistic personal-assistant workflows
- Testing communication, finance, operations and productivity task execution
- Evaluating multimodal webpage generation, video question answering and document extraction
- Evaluating multi-turn clarification and professional advice interactions
- Auditing safety violations and robustness failures across repeated agent runs
- Comparing frontier and open models before deployment in agentic software environments