# agent-eval (is-it-agentic-enough)

> Benchmarks how coding agents use a library across revisions and models, producing a static HTML report publishable as a Hugging Face Space.

agent-eval is an open-source Python tool from Hugging Face that measures how a coding agent uses a library, not only whether it reaches the right answer. It runs the same tasks across library revisions and models, then renders the results as a static HTML report that can be hosted as a Hugging Face Space.

## What It Is

agent-eval is an evaluation harness for agent-friendliness of libraries. It tracks how an agent gets to an answer: whether it reaches for a CLI or hand-writes Python, how many tokens and seconds it spends, and how often it errors. Environment-specific details, including the task suite, live behind a profile. The `transformers` profile is the reference study, and a `mock` profile runs no agent and is meant for trying the UI quickly.

## How the Workflow Runs

You declare a YAML matrix of profile, tasks, runs per cell, hardware flavor, open models served on Hugging Face inference providers, and library revisions. The `batch` command prints a dry-run plan until `--submit` is passed, then launches one Hugging Face Job per model and revision, or per task with `--per-task`. Each revision is run across three tiers of assistance: bare, clone (repo in the working directory), and skill (a packaged Skill). Every run records its transcript, metadata, and native agent session. Runs are persisted to a shared Hugging Face bucket as they finish, and re-running the same file skips completed cells.

## The Report

The report is a single self-contained, theme-aware page with three tabs. Overview shows match percentage, median time, median tokens, and error percentage, plus label adoption such as CLI versus `pipeline()` and per-run distributions. Coverage is a task by revision heatmap of done versus expected runs. Results lists every task and what each model answered, with click-through into failing responses.

## Safety Notes

The `transformers` profile runs a coding agent with bypassed permissions and executes code from the revision you point it at, and traces can contain prompts, output, and paths. The README advises trusted local use only; the `mock` profile runs no agent and is described as always safe.

## Features
- Measures match rate, median time, median tokens, and error rate of coding agents
- Compares library revisions and models in a YAML matrix
- Three assistance tiers: bare, clone, and skill
- Launches runs as Hugging Face Jobs with dry-run by default
- Persists results to a Hugging Face bucket and skips completed cells on rerun
- Static HTML report with Overview, Coverage, and Results tabs
- Publish report as a Hugging Face Space
- Package native agent traces into a Hugging Face dataset
- Profile system with transformers and mock profiles

## Integrations
Hugging Face Jobs, Hugging Face Spaces, Hugging Face Hub buckets, Hugging Face datasets, Hugging Face inference providers, transformers

## Platforms
API, CLI

## Pricing
Open Source

## Links
- Website: https://github.com/huggingface/is-it-agentic-enough
- Repository: https://github.com/huggingface/is-it-agentic-enough
- EveryDev.ai: https://www.everydev.ai/tools/agent-eval