agent-eval from cryer
Open-source CI tool that replays eval suites for LLM agents and uses statistical tests to block regressions.
At a Glance
Self-hosted agent-eval core released under the MIT License; free to use, copy, modify, and distribute.
Engagement
Available On
Alternatives
Listed Nov 2026
About agent-eval from cryer
agent-eval is an open-source Python tool from the GitHub user cryer that works as continuous integration for LLM agents. It replays an evaluation suite against a baseline and a candidate agent, then uses a clustered bootstrap gate to decide whether a change made behavior worse or whether the difference is noise. It is released under the MIT license.
What It Is
agent-eval is an evaluation and regression-gating framework for agents whose behavior is probabilistic, multi-turn and tool-driven. A change to a prompt, model version, tool schema or agent loop can be replayed as baseline versus candidate, and the result becomes a CI pass, regression or inconclusive verdict. Runs are traced with OpenTelemetry spans (LLM calls, tool calls, token cost, prompt version hash) and persisted to SQLite, and trajectory diffs show where behavior changed.
How the Evaluation Works
Assertions come in two layers: deterministic rules (output content, tool usage, turn/cost/latency budgets, workspace state) and LLM-as-judge checks for semantic quality. Simulated users with hidden goals run multi-turn rollouts against personas, including adversarial ones such as a user fishing for the system prompt. Coding-agent cases can carry a repo-snapshot fixture, with each sample run in a fresh copy and final workspace state checked through assertions like command exit and diff size limits. Judges can be admitted through calibration against human labels, using accuracy, Cohen's kappa, FPR and FNR.
The Statistical CI Gate
The compare step bootstraps whole cases rather than correlated samples. With a minimum important regression margin (default 0.05) and a confidence interval, it returns pass, regression or inconclusive, and too few cases is also inconclusive. Manifests recording Git, dataset, model and judge configuration are validated before any statistics run. Exit codes are defined for run, simulate, compare and calibrate. The included GitHub Actions workflow runs the target branch as baseline and the PR as candidate, and uploads JSON and JUnit XML reports, trajectory diffs and the SQLite database as artifacts.
Setup Path
Install with pip from the repository; an optional llm extra enables real-model judges and agents through LiteLLM-supported providers. Any callable can be evaluated, and OTel instrumentation unlocks cost, tool and trajectory assertions. A distill command can generate new cases from recorded traces, and concurrency with exponential backoff handles 429/5xx rate limits.
Community Discussions
Be the first to start a conversation about agent-eval from cryer
Share your experience with agent-eval from cryer, ask questions, or help others learn from your insights.
Pricing
Open Source (MIT)
Self-hosted agent-eval core released under the MIT License; free to use, copy, modify, and distribute.
- MIT-licensed core CLI/library
- OpenTelemetry tracing persisted to SQLite
- Deterministic rule assertions and LLM-as-judge
- Judge calibration admission
- Simulated users with hidden goals
Capabilities
Key Features
- OpenTelemetry tracing persisted to SQLite
- Trajectory diffs between baseline and candidate
- Deterministic rule assertions plus LLM-as-judge
- Judge calibration admission with kappa, FPR and FNR
- Simulated users with hidden goals and adversarial personas
- Coding-agent evaluation with repo-snapshot fixtures
- Clustered bootstrap statistical regression gate
- Run manifests and compatibility checks
- JSON and JUnit XML CI reports
- Parallel runs with exponential backoff on 429/5xx
- Case distillation from recorded traces
- GitHub Actions workflow
