smevals

smevals is an open-source Python framework from Prime Radiant, Inc. for running evals against small and large models. It organizes work into Evals, Tasks, Configs, Runs, Graders and Checks, and is installed as a CLI tool. The latest listed release is 0.2.0.

What It Is

smevals is a command-line evaluation harness. An Eval is a directory containing an eval.yaml file, with YAML files for Tasks, Configs and Graders. Runner executables send prompts to a model directly or drive an agent harness such as Codex or Pi. Checker programs can also be custom executables.

How the Workflow Operates

A Run records one Task executed against one Config through a Runner. The Runner receives its inputs as environment variables such as SMEVALS_MODEL and SMEVALS_PROMPT. Its standard output is saved as output.txt, and a non-zero exit marks the Run as failed. Failed Runs are never graded and do not count toward the -n sample target.

Graders apply an ordered sequence of Checks to each Run. Built-in checkers include contains and xml-valid, and custom checkers can emit scores, metrics, tags and notes. Checks can be marked required, and files created by one Check are available to later Checks. Each Grade stores a snapshot of its Grader. Runs can be regraded without being re-executed, and multiple Graders can coexist.

Reporting and Viewing Results

smevals report prints a markdown leaderboard of config and model combinations with mean and standard error scores. It also shows tag shares and per-task breakdowns. smevals serve provides a live web UI, and smevals build generates a static site that can aggregate Evals from multiple repositories.