# smevals

> A command-line framework for running and grading evals against small and large language models.

smevals is an open-source Python framework from Prime Radiant, Inc. for running evals against small and large models. It organizes work into Evals, Tasks, Configs, Runs, Graders and Checks, and is installed as a CLI tool. The latest listed release is 0.2.0.

## What It Is

smevals is a command-line evaluation harness. An Eval is a directory containing an `eval.yaml` file, with YAML files for Tasks, Configs and Graders. Runner executables send prompts to a model directly or drive an agent harness such as Codex or Pi. Checker programs can also be custom executables.

## How the Workflow Operates

A Run records one Task executed against one Config through a Runner. The Runner receives its inputs as environment variables such as `SMEVALS_MODEL` and `SMEVALS_PROMPT`. Its standard output is saved as `output.txt`, and a non-zero exit marks the Run as failed. Failed Runs are never graded and do not count toward the `-n` sample target.

Graders apply an ordered sequence of Checks to each Run. Built-in checkers include `contains` and `xml-valid`, and custom checkers can emit scores, metrics, tags and notes. Checks can be marked required, and files created by one Check are available to later Checks. Each Grade stores a snapshot of its Grader. Runs can be regraded without being re-executed, and multiple Graders can coexist.

## Reporting and Viewing Results

`smevals report` prints a markdown leaderboard of config and model combinations with mean and standard error scores. It also shows tag shares and per-task breakdowns. `smevals serve` provides a live web UI, and `smevals build` generates a static site that can aggregate Evals from multiple repositories.

## Features
- Eval, Task, Config, Run, Grader and Check model
- Runner contract using environment variables
- Built-in contains and xml-valid checkers
- Custom checker executables
- Repeated runs with -n sample targets
- Regrading without re-running
- Markdown leaderboard reports with mean ± stderr
- Live web UI via serve
- Static site build
- Multiple graders per eval

## Integrations
llm CLI, Codex, Pi

## Platforms
API, CLI

## Pricing
Open Source

## Version
0.2.0

## Links
- Website: https://github.com/prime-radiant-inc/smevals
- Documentation: https://github.com/prime-radiant-inc/smevals
- Repository: https://github.com/prime-radiant-inc/smevals
- EveryDev.ai: https://www.everydev.ai/tools/smevals