EveryDev.ai
Subscribe
Home
Tools

5,923+ AI tools

  • Featured
  • Gems
  • Trending
  • New
  • Rate tools
  • Compare
  • Arena
Categories
  • Agents2992
  • Coding2260
  • Infrastructure1068
  • Projects824
  • MCP795
  • Research744
  • Analytics567
  • Marketing527
  • Design489
  • Testing476
  • Data442
  • Security417
  • Integration288
  • Communication286
  • Prompts284
  • Extensions239
  • Voice214
  • DevOps191
  • Commerce184
  • Web128
  • Learning123
  • Finance44
AI Tools by Topic
  • AI Coding Assistants
  • Agent Frameworks
  • MCP Servers
  • AI Prompt Tools
  • Vibe Coding Tools
  • AI Design Tools
  • AI Database Tools
  • AI Website Builders
  • AI Testing Tools
  • LLM Evaluations
Follow Us
  • X / Twitter
  • LinkedIn
  • Reddit
  • Discord
  • Threads
  • Bluesky
  • Mastodon
  • YouTube
  • GitHub
  • Instagram
Get Started
  • Users
  • Rate Tools
  • About
  • Editorial Standards
  • Corrections & Disclosures
  • Community Guidelines
  • Advertise
  • Contact Us
  • Newsletter
  • Submit a Tool
  • Start a Discussion
  • Write A Blog
  • Share A Build
  • Terms of Service
  • Privacy Policy
Explore with AI
  • ChatGPT
  • Gemini
  • Claude
  • Grok
  • Perplexity
Agent Experience
  • llms.txt
Theme
With AI, Everyone is a Dev. EveryDev.ai ©
    1. Home
    2. Tools
    3. agent-eval from cryer
    agent-eval from cryer icon

    agent-eval from cryer

    Automated Testing

    Open-source CI tool that replays eval suites for LLM agents and uses statistical tests to block regressions.

    Visit Website

    At a Glance

    Pricing
    Open Source

    Self-hosted agent-eval core released under the MIT License; free to use, copy, modify, and distribute.

    Engagement

    Available On

    API
    CLI

    Resources

    WebsiteDocsGitHubllms.txt

    Topics

    Automated TestingCI/CD ToolsLLM Evaluations

    Alternatives

    RegentAgentBenchAgencyBench
    Developer
    kurumiNanjing, China

    Listed Nov 2026

    About agent-eval from cryer

    agent-eval is an open-source Python tool from the GitHub user cryer that works as continuous integration for LLM agents. It replays an evaluation suite against a baseline and a candidate agent, then uses a clustered bootstrap gate to decide whether a change made behavior worse or whether the difference is noise. It is released under the MIT license.

    What It Is

    agent-eval is an evaluation and regression-gating framework for agents whose behavior is probabilistic, multi-turn and tool-driven. A change to a prompt, model version, tool schema or agent loop can be replayed as baseline versus candidate, and the result becomes a CI pass, regression or inconclusive verdict. Runs are traced with OpenTelemetry spans (LLM calls, tool calls, token cost, prompt version hash) and persisted to SQLite, and trajectory diffs show where behavior changed.

    How the Evaluation Works

    Assertions come in two layers: deterministic rules (output content, tool usage, turn/cost/latency budgets, workspace state) and LLM-as-judge checks for semantic quality. Simulated users with hidden goals run multi-turn rollouts against personas, including adversarial ones such as a user fishing for the system prompt. Coding-agent cases can carry a repo-snapshot fixture, with each sample run in a fresh copy and final workspace state checked through assertions like command exit and diff size limits. Judges can be admitted through calibration against human labels, using accuracy, Cohen's kappa, FPR and FNR.

    The Statistical CI Gate

    The compare step bootstraps whole cases rather than correlated samples. With a minimum important regression margin (default 0.05) and a confidence interval, it returns pass, regression or inconclusive, and too few cases is also inconclusive. Manifests recording Git, dataset, model and judge configuration are validated before any statistics run. Exit codes are defined for run, simulate, compare and calibrate. The included GitHub Actions workflow runs the target branch as baseline and the PR as candidate, and uploads JSON and JUnit XML reports, trajectory diffs and the SQLite database as artifacts.

    Setup Path

    Install with pip from the repository; an optional llm extra enables real-model judges and agents through LiteLLM-supported providers. Any callable can be evaluated, and OTel instrumentation unlocks cost, tool and trajectory assertions. A distill command can generate new cases from recorded traces, and concurrency with exponential backoff handles 429/5xx rate limits.

    agent-eval from cryer - 1

    Community Discussions

    Be the first to start a conversation about agent-eval from cryer

    Share your experience with agent-eval from cryer, ask questions, or help others learn from your insights.

    Pricing

    OPEN SOURCE

    Open Source (MIT)

    Self-hosted agent-eval core released under the MIT License; free to use, copy, modify, and distribute.

    • MIT-licensed core CLI/library
    • OpenTelemetry tracing persisted to SQLite
    • Deterministic rule assertions and LLM-as-judge
    • Judge calibration admission
    • Simulated users with hidden goals

    Capabilities

    Key Features

    • OpenTelemetry tracing persisted to SQLite
    • Trajectory diffs between baseline and candidate
    • Deterministic rule assertions plus LLM-as-judge
    • Judge calibration admission with kappa, FPR and FNR
    • Simulated users with hidden goals and adversarial personas
    • Coding-agent evaluation with repo-snapshot fixtures
    • Clustered bootstrap statistical regression gate
    • Run manifests and compatibility checks
    • JSON and JUnit XML CI reports
    • Parallel runs with exponential backoff on 429/5xx
    • Case distillation from recorded traces
    • GitHub Actions workflow

    Integrations

    GitHub Actions
    OpenTelemetry
    SQLite
    LiteLLM
    JUnit XML
    Kimi K3
    API Available
    View Docs

    Ratings & Reviews

    No ratings yet

    Be the first to rate agent-eval from cryer and help others make informed decisions.

    Rate other tools you’ve used

    Developer

    kurumi

    Major in machine learning and deep learning and a little computer graphics.

    Nanjing, China
    Read more about kurumi
    WebsiteGitHub
    1 tool in directory

    Similar Tools

    Regent icon

    Regent

    Regent proxies every LLM call in your app to detect behavioral regressions between code versions, posting detailed diff reports directly to pull requests.

    AgentBench icon

    AgentBench

    AgentBench is an open-source benchmark framework for evaluating LLMs as autonomous agents across 8 diverse environments including OS, database, web, and knowledge graph tasks.

    AgencyBench icon

    AgencyBench

    Open-source benchmark that evaluates autonomous LLM agents on long-horizon, real-world tasks with automated rubric-based scoring.

    Browse all tools

    Related Topics

    Automated Testing

    AI-powered platforms that automate end-to-end testing processes with intelligent test case generation, execution, and reporting for faster, more reliable software delivery.

    208 tools

    CI/CD Tools

    AI-powered continuous integration and continuous deployment platforms that automate testing, building, and releasing software with intelligent insights and optimization.

    62 tools

    LLM Evaluations

    Platforms and frameworks for evaluating, testing, and benchmarking LLM systems and AI applications. These tools provide evaluators and evaluation models to score AI outputs, measure hallucinations, assess RAG quality, detect failures, and optimize model performance. Features include automated testing with LLM-as-a-judge metrics, component-level evaluation with tracing, regression testing in CI/CD pipelines, custom evaluator creation, dataset curation, and real-time monitoring of production systems. Teams use these solutions to validate prompt effectiveness, compare models side-by-side, ensure answer correctness and relevance, identify bias and toxicity, prevent PII leakage, and continuously improve AI product quality through experiments, benchmarks, and performance analytics.

    242 tools
    Browse all topics
    Back to all toolsSuggest an edit
    ratings
    discussions
    13views