# Letta Evals

> Open-source evaluation framework for testing and grading Letta and Letta Code stateful agents using YAML suites, datasets, graders and rewards.

Letta Evals is an evaluation framework from Letta for testing Letta and Letta Code agents. Users define a suite with a dataset, target, extractors, graders and a reward contract, then run it against one or more model configurations from the command line. The project is Apache 2.0 licensed and distributed on PyPI as letta-evals.

## What It Is

Letta Evals is a testing toolkit for stateful agents. The docs describe it as a way to validate changes, catch regressions and test agent memory, tool usage, multi-turn conversations and state evolution with automated grading and pass/fail gates. Suites are written in YAML with JSONL or CSV datasets, so many evals need no Python code.

## How an Evaluation Runs

The core flow is Dataset, Target, Extractor, Grader, Reward, Result. The target runs the Letta Code CLI against a Letta server, either self-hosted or Letta Cloud. Extractors pull content from agent responses, such as the last assistant message. Graders score that content, either deterministically with tool graders (for example contains, exact_match, regex_match) or with model-judge rubrics. A reward turns grader outputs into a per-sample score, and custom Python reward composers can combine multiple graders.

## Iteration and Scale

Suites can evaluate several model handles in one run, repeat runs for aggregate statistics, and re-grade saved trajectories with a cached results file without re-running the target. Multi-turn datasets support per-turn grading. Datasets can also be loaded from the HuggingFace Hub. An optional Modal sandbox setting runs each sample in a fresh sandbox with pinned versions. The README says it is designed for CI use, with a GitHub Actions workflow as an example.

## Extensibility

Custom graders, extractors, setup hooks and agent factories can be registered in Python, and per-sample rubric overrides are supported. The repository includes example suites for custom graders, reward composition, multi-model comparison and programmatic agent creation.

## Features
- YAML-defined evaluation suites
- JSONL and CSV datasets, including HuggingFace Hub datasets
- Deterministic tool graders and LLM model-judge graders
- Built-in and custom extractors
- Custom reward composers
- Multi-model comparison in a single suite
- Multi-turn conversations with per-turn grading
- Repeated runs with aggregate statistics
- Re-grading cached trajectories
- Modal sandbox execution per sample
- CLI with validate, run, list-graders and list-extractors commands
- CI/CD support

## Integrations
Letta, Letta Code, Letta Cloud, HuggingFace Hub, Modal, GitHub Actions, OpenAI, Anthropic, Google

## Platforms
WEB, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
0.26.0

## Links
- Website: https://docs.letta.com/evals
- Documentation: https://docs.letta.com/v1-sdk/evals/
- Repository: https://github.com/letta-ai/letta-evals
- EveryDev.ai: https://www.everydev.ai/tools/letta-evals