# OpenEvals

> Open-source Python and TypeScript library of readymade evaluators for testing LLM applications, including LLM-as-judge, structured output, code, and agent trajectory evals.

OpenEvals is an open-source evaluation library from LangChain that provides readymade evaluators for LLM applications, available in Python and TypeScript. It gives developers a starting point for writing evals, which they can then customize for their own application. The latest release listed in the repository is openevals-js==0.2.2.

## What It Is

OpenEvals is a package of evaluator functions that score the outputs of LLM-based systems, much like tests in traditional software. Developers pass in `inputs`, `outputs`, and optionally `reference_outputs`, and receive a result containing a key, a score, and a comment. Evaluators specific to LLM agents are covered by a separate LangChain package, `agentevals`, which the README points to.

## LLM-as-Judge and Prebuilt Prompts

The core evaluator is `create_llm_as_judge`, which takes a prompt and a model and returns an evaluator function that formats parameters into the prompt and parses the judge model's output into a score. Prompts can be f-strings, LangChain prompt templates, or functions that return messages. Scores can be binary, continuous, or drawn from custom choices, and custom output schemas are supported. The judge model can be set via a `PROVIDER:MODEL` string, a LangChain chat model, or an OpenAI client.

Prebuilt prompts cover several areas:
- Quality: conciseness, correctness, hallucination, answer relevance, plan adherence, code correctness, and laziness.
- Safety and security: toxicity, fairness, PII leakage, prompt injection, and code injection.
- Multimodal: image prompts such as explicit content and sensitive imagery, and voice prompts such as audio quality and transcription accuracy, using image, audio, and PDF attachments.
- RAG: correctness, helpfulness, groundedness, and retrieval relevance.

## Structured Output, Code, and Agent Trajectories

A JSON match evaluator checks extraction and tool-call outputs with exact match or with LLM-as-judge rubrics for chosen keys, with aggregation across keys and lists. Code evaluators extract code from LLM output and type-check it with Pyright, Mypy, or the TypeScript compiler, or judge it with an LLM. Sandboxed variants use E2B to install dependencies and run Pyright, TypeScript type-checking, or execution in isolation.

For agents, trajectory match evaluators compare tool-call sequences against a reference in strict, unordered, subset, or superset modes, with configurable tool-argument matching. A trajectory LLM-as-judge assesses accuracy with or without a reference. Other string evaluators include exact match, Levenshtein distance, and embedding similarity.

## Setup and Integrations

Install with `pip install openevals` for Python or `npm install openevals @langchain/core` for TypeScript. LLM-as-judge evaluators need an LLM client, by default through LangChain chat model integrations. The README also describes multiturn simulation with simulated users and LangGraph, and LangSmith integration through pytest, Vitest/Jest, or `evaluate`.

## Features
- LLM-as-judge evaluator factory with customizable prompts, models, and scoring
- Prebuilt prompts for quality, safety, security, image, voice, and RAG evaluation
- Structured output and tool call evaluation with JSON match evaluator
- Code evaluators using Pyright, Mypy, TypeScript type-checking, and LLM-as-judge
- Sandboxed code type-checking and execution via E2B
- Agent trajectory match evaluators in strict, unordered, subset, and superset modes
- Trajectory LLM-as-judge evaluator
- Exact match, Levenshtein distance, and embedding similarity evaluators
- Multimodal support for images, audio, and PDFs
- Multiturn simulation with simulated users
- Custom output schemas and custom evaluator creation
- Python async support

## Integrations
LangChain, LangSmith, LangGraph, OpenAI, Anthropic, Google Gemini, E2B, Pyright, Mypy, Pytest, Vitest, Jest

## Platforms
WINDOWS, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
openevals-js==0.2.2

## Links
- Website: https://github.com/langchain-ai/openevals
- Repository: https://github.com/langchain-ai/openevals
- EveryDev.ai: https://www.everydev.ai/tools/openevals