Verifiers

Verifiers is an open-source Python library developed by Prime Intellect AI for building environments that train and evaluate large language models (LLMs) via reinforcement learning (RL). Originally created by Will Brown, the project is hosted on GitHub under the MIT License and integrates tightly with Prime Intellect's broader platform, including the Environments Hub, the prime-rl training framework, and Hosted Training infrastructure.

What It Is

Verifiers provides a structured way to define self-contained RL environments for LLMs. Each environment bundles three core components: a dataset of task inputs, a harness for the model (covering tools, sandboxes, and context management), and a rubric (reward function) to score model performance. These environments can be used for RL training, capability evaluation, synthetic data generation, and agent harness experimentation.

Architecture and Core Concepts

The library supports both a legacy single-turn API and a newer v1 Taskset/Harness API introduced in v0.1.14. Key abstractions include:

• Taskset: Defines the dataset rows, reward functions, and split configuration for a task.
• Harness: Manages how the model interacts with tools, sandboxes, and multi-turn context.
• Rubric: Scores model completions using async reward functions with configurable weights.
• Environment types: SingleTurnEnv, MultiTurnEnv, RLMEnv, CliAgentEnv, BrowserEnv, and OpenEnv integrations cover a wide range of interaction protocols.

Environments are self-contained Python modules installable via uv, and the same package is used for both evals and RL training runs.

Workflow and Tooling

The prime CLI drives the development workflow end-to-end:

• prime lab setup scaffolds a workspace with recommended configs for training, eval, and prompt optimization (GEPA).
• prime env init generates a new environment template.
• prime eval run executes local evaluations against any OpenAI-compatible endpoint.
• prime env push publishes environments to the public Environments Hub.
• vf-tui provides a terminal UI for reviewing eval results.

Configuration is TOML-based, with separate config shapes for RL training (model, batch size, rollouts) and environment-specific options (taskset split, harness max turns, reward weights).

Environments Hub Integration

Verifiers connects to the Prime Intellect Environments Hub, a shared registry where community-built environments can be published, discovered, and installed. Environments from the hub (e.g., primeintellect/math-python) can be installed directly into a local project and used for both evaluation and training without modification.

Update: v0.1.14 and Active Development

The project has seen rapid release cadence since its creation in January 2025. As of the latest available data:

• v0.1.15.dev7 is the most recent pre-release (published May 15, 2026).
• v0.1.14 (released May 7, 2026) introduced the v1 Taskset/Harness API, shared eval/training config shapes, model-family starter configs, OpenAI Responses and renderer-backed clients, per-turn timing, GEPA prompt artifacts, Lean guard markers, and infrastructure hardening.
• Earlier notable releases include v0.1.12 (RLMEnv improvements, multi-worker env server), v0.1.11 (unified client stack, eval TUI), v0.1.10 (OpenEnv and BrowserEnv integrations), and v0.1.8 (trajectory-based rollout tracking for token-in/token-out training).

The repository reports 4,123 stars and 551 forks as of the last recorded update, according to GitHub metadata.