agent-eval (is-it-agentic-enough)

agent-eval is an open-source Python tool from Hugging Face that measures how a coding agent uses a library, not only whether it reaches the right answer. It runs the same tasks across library revisions and models, then renders the results as a static HTML report that can be hosted as a Hugging Face Space.

What It Is

agent-eval is an evaluation harness for agent-friendliness of libraries. It tracks how an agent gets to an answer: whether it reaches for a CLI or hand-writes Python, how many tokens and seconds it spends, and how often it errors. Environment-specific details, including the task suite, live behind a profile. The transformers profile is the reference study, and a mock profile runs no agent and is meant for trying the UI quickly.

How the Workflow Runs

You declare a YAML matrix of profile, tasks, runs per cell, hardware flavor, open models served on Hugging Face inference providers, and library revisions. The batch command prints a dry-run plan until --submit is passed, then launches one Hugging Face Job per model and revision, or per task with --per-task. Each revision is run across three tiers of assistance: bare, clone (repo in the working directory), and skill (a packaged Skill). Every run records its transcript, metadata, and native agent session. Runs are persisted to a shared Hugging Face bucket as they finish, and re-running the same file skips completed cells.

The Report

The report is a single self-contained, theme-aware page with three tabs. Overview shows match percentage, median time, median tokens, and error percentage, plus label adoption such as CLI versus pipeline() and per-run distributions. Coverage is a task by revision heatmap of done versus expected runs. Results lists every task and what each model answered, with click-through into failing responses.

Safety Notes

The transformers profile runs a coding agent with bypassed permissions and executes code from the revision you point it at, and traces can contain prompts, output, and paths. The README advises trusted local use only; the mock profile runs no agent and is described as always safe.