EvalScope

EvalScope is an open-source evaluation framework from the ModelScope Community. It runs benchmark evaluations of LLMs, VLMs, embedding, reranker and AIGC models, load-tests model services, and produces reports and a web dashboard. The latest release listed on GitHub is v1.12.0.

What It Is

EvalScope is a command-line and Python framework for measuring AI systems. A single command such as evalscope eval points at an OpenAI-compatible API or a local model (auto-downloaded from ModelScope), runs selected datasets like GSM8K or ARC, and prints a score table. Results are written to an outputs directory with logs, configs, predictions, reviews and HTML reports.

Evaluation and Benchmarks

The project's site says its registry holds 250+ benchmarks across LLM, VLM, agent and AIGC tasks. Beyond the native backend, it can launch OpenCompass, VLMEvalKit and RAG evaluation backends. Other documented modes include Arena Mode, which ranks models through pairwise battles, custom dataset evaluation, and System One Choice evaluation. Metric aggregation such as pass@k is also documented.

Agent and Harness Evaluation

Agent evaluation drives benchmarks through a multi-turn agent loop with pluggable strategies, tools such as bash and Python execution, and local or Docker sandbox environments. Each sample records an agent trace of model requests, tool calls and outcomes. An External Agent Bridge can evaluate off-the-shelf agent CLIs like Claude Code and Codex, and MCP servers can be attached as tools.

Performance Testing and Visualization

The evalscope perf tool measures concurrency, latency, TTFT, TPOT, RPS, throughput and success rate against a model endpoint. It supports multi-turn stress tests and embedding and reranker services. Results can be sent to wandb, swanlab or ClearML. evalscope service starts a local dashboard for model comparison, report overviews and prediction details.