OpenEvals

OpenEvals is an open-source evaluation library from LangChain that provides readymade evaluators for LLM applications, available in Python and TypeScript. It gives developers a starting point for writing evals, which they can then customize for their own application. The latest release listed in the repository is openevals-js==0.2.2.

What It Is

OpenEvals is a package of evaluator functions that score the outputs of LLM-based systems, much like tests in traditional software. Developers pass in inputs, outputs, and optionally reference_outputs, and receive a result containing a key, a score, and a comment. Evaluators specific to LLM agents are covered by a separate LangChain package, agentevals, which the README points to.

LLM-as-Judge and Prebuilt Prompts

The core evaluator is create_llm_as_judge, which takes a prompt and a model and returns an evaluator function that formats parameters into the prompt and parses the judge model's output into a score. Prompts can be f-strings, LangChain prompt templates, or functions that return messages. Scores can be binary, continuous, or drawn from custom choices, and custom output schemas are supported. The judge model can be set via a PROVIDER:MODEL string, a LangChain chat model, or an OpenAI client.

Prebuilt prompts cover several areas:

• Quality: conciseness, correctness, hallucination, answer relevance, plan adherence, code correctness, and laziness.
• Safety and security: toxicity, fairness, PII leakage, prompt injection, and code injection.
• Multimodal: image prompts such as explicit content and sensitive imagery, and voice prompts such as audio quality and transcription accuracy, using image, audio, and PDF attachments.
• RAG: correctness, helpfulness, groundedness, and retrieval relevance.

Structured Output, Code, and Agent Trajectories

A JSON match evaluator checks extraction and tool-call outputs with exact match or with LLM-as-judge rubrics for chosen keys, with aggregation across keys and lists. Code evaluators extract code from LLM output and type-check it with Pyright, Mypy, or the TypeScript compiler, or judge it with an LLM. Sandboxed variants use E2B to install dependencies and run Pyright, TypeScript type-checking, or execution in isolation.

For agents, trajectory match evaluators compare tool-call sequences against a reference in strict, unordered, subset, or superset modes, with configurable tool-argument matching. A trajectory LLM-as-judge assesses accuracy with or without a reference. Other string evaluators include exact match, Levenshtein distance, and embedding similarity.

Setup and Integrations

Install with pip install openevals for Python or npm install openevals @langchain/core for TypeScript. LLM-as-judge evaluators need an LLM client, by default through LangChain chat model integrations. The README also describes multiturn simulation with simulated users and LangGraph, and LangSmith integration through pytest, Vitest/Jest, or evaluate.