AgentEvals

AgentEvals is an open-source package from LangChain that provides evaluators and utilities for judging the intermediate steps an agent takes as it runs, known as its trajectory. It is available for Python and TypeScript and is released under the MIT License. The README positions it as a conceptual starting point for agent evals, with the companion package openevals covering more general evaluation.

What It Is

AgentEvals is an agent evaluation library. You represent an agent's run as a list of OpenAI-style messages (or LangChain message objects) and pass it to an evaluator, which returns a keyed result with a score and, for LLM judges, a comment explaining the reasoning. Evaluators can be used with or without a reference trajectory, depending on the type.

Evaluator Types

Trajectory match evaluators compare an agent's tool calls with a reference trajectory in four modes: strict (same messages and tool calls in the same order), unordered (same tool calls in any order), subset, and superset. Tool-argument matching can be set to exact, ignore, subset, or superset, with per-tool overrides that accept a list of fields or a custom comparator function.

The trajectory LLM-as-judge evaluator uses a judge model with a prebuilt accuracy prompt, with or without a reference. It accepts options such as continuous scoring, custom choices, a system prompt, and few-shot examples.

Graph Trajectories and LangGraph

For graph-based agents such as LangGraph, graph trajectory evaluators work on nodes visited rather than messages. Utilities extract a trajectory from a LangGraph thread, including multiple turns and interrupts. A graph trajectory LLM-as-judge evaluator and a strict-match evaluator are provided.

Setup and Integrations

Install with pip install agentevals or npm install agentevals @langchain/core. LLM-as-judge evaluators use LangChain chat model integrations by default, or the OpenAI client directly. Python async variants exist for all evaluators. Results can be logged to LangSmith through its pytest or Vitest/Jest integrations or its evaluate function.