Judgeval

Judgeval is an open-source Python SDK from Judgment Labs that adds tracing and agent-judge evaluation to LLM-powered applications. It is part of the Judgment platform, which the company describes as the continuous-improvement stack for agents: detect failures, triage root causes, and validate fixes against production data. The SDK is released under the Apache 2.0 license.

What It Is

Judgeval instruments agent code so that inputs, outputs, and LLM token usage are captured as traces. Tracing is built on OpenTelemetry, so it is designed to work with existing observability stacks. Developers decorate functions with @Tracer.observe() and wrap supported LLM clients to start collecting data.

On top of the traces, users define prompt-based agent judges that score agent behaviors. Judges produce structured, labeled behaviors that accumulate into a searchable record of how the agent acted over time.

How the Workflow Fits Together

Judges can run against live production traffic, scoring it server-side, or be replayed on historical traces to test a proposed fix before shipping. Detected behaviors can trigger Slack alerts so regressions and recurrences get surfaced. The Judgment platform shown on the homepage also demonstrates investigating issues from Slack, triaging similar failure cases across traces, and running agent tests built from production cases.

The SDK also exposes read-only SQL queries against Judgment's virtual schema, with schema discovery, organization and project scoping, and result limits of 1,000 rows and 5 MiB.

Integrations and Access Points

Auto-instrumentation covers OpenAI, Anthropic, Google GenAI, and Together AI, with framework support for LangGraph, OpenLit, and the Claude Agent SDK. A separate CLI manages agents, traces, judges, behaviors, and evaluations from the terminal. An MCP server lets MCP-compatible AI tools query traces, invoke judges, and browse detected behaviors.

Research Focus

Judgment Labs describes itself as an applied-research lab working on production agent reliability. It lists productionized research areas including Agent Search, Agent Judge, Behavior Discovery, and AutoRubrics.