# LLM-as-a-Verifier

> Open-source Python framework that gives fine-grained, logit-based verification scores for best-of-N selection, pairwise comparison, and progress tracking of agent trajectories.

LLM-as-a-Verifier is an open-source Python framework from the LLM-as-a-Verifier team, with authors listed from Stanford, UC Berkeley and NVIDIA Research. It scores agent trajectories using the full distribution of scoring-token logits, so no additional training is needed. The scores can drive best-of-N selection, step-by-step progress tracking, and dense rewards for reinforcement learning.

## What It Is

The library is installed with `pip install llm-verifier` and exposes three calls: `select` for best-of-N selection, `compare` for pairwise scoring, and `track` for progress scoring over a rollout. Instead of asking an LLM judge for a single discrete score, it takes the expectation over the logprob distribution of score tokens on a 1-20 scale. This yields continuous rewards in [0, 1] and, according to the project, avoids the ties that coarse judges produce. Verification can also be scaled by repeating evaluations and by decomposing the rubric into several criteria.

## How Selection and Tracking Work

For best-of-N selection, the Probabilistic Pivot Tournament ranks candidates by comparing them against a small set of pivots. This takes O(Nk) pairwise verifications instead of a full O(N²) round-robin. A ring pass first scores adjacent pairs to cancel positional bias, and the top candidates then become pivots. For progress tracking, `track` scores a finished trajectory, while `ProgressTracker` returns a live score after each step of a running agent. Every entry point accepts optional images, which supports VLM agents and robotics rollouts.

## Benchmarks and Reproduction

The repository ships agent trajectories and scripts to reproduce results on Terminal-Bench, SWE-Bench Verified and MedAgentBench. The project reports 86.5% on Terminal-Bench V2, 78.2% on SWE-Bench Verified, 73.3% on MedAgentBench and 87.4% on RoboRewardBench. It also reports improved sample efficiency when the verifier is used as a dense reward for SAC and GRPO. Users can add their own benchmark through criteria files and a documented workflow.

## Setup Path

A verifier backend is needed: a DeepSeek or Vertex API key in `.env`, or an OpenAI-compatible server that returns logprobs, such as vLLM. A companion Claude Code plugin, TurboAgent, runs as an LLM API proxy that generates multiple candidate responses and selects the best one. Version 0.3.0 adds a Terminal-Bench 4.0 benchmark, task-agnostic criteria and an updated DeepSeek backend.

## Features
- Best-of-N selection with llm_verifier.select
- Pairwise comparison with llm_verifier.compare
- Progress tracking with track and online ProgressTracker
- Fine-grained continuous rewards from score-token logprobs
- Probabilistic Pivot Tournament with O(Nk) comparisons
- Criteria decomposition and repeated evaluation
- Multimodal image and video inputs
- Dense reward signals for reinforcement learning
- Benchmark reproduction scripts and custom benchmark support
- Token usage accounting and prefix-cache optimization

## Integrations
Claude Code (TurboAgent plugin), Codex, DeepSeek, Google Vertex AI / Gemini, OpenAI-compatible servers (vLLM)

## Platforms
LINUX, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
0.3.0

## Links
- Website: https://llm-as-a-verifier.com/docs/
- Documentation: https://llm-as-a-verifier.com/docs/
- Repository: https://github.com/llm-as-a-verifier/llm-as-a-verifier
- EveryDev.ai: https://www.everydev.ai/tools/llm-as-a-verifier