LLM-as-a-Verifier Team
LLM-as-a-Verifier is a research project and open-source framework, rather than a conventional incorporated company. It provides fine-grained, training-free feedback for agent trajectories and other modalities by taking expectations over scoring-token logit distributions; the feedback supports best-of-N/test-time selection, progress tracking, and reinforcement learning.
At a Glance
- AI-agent developers and engineering teams
- Researchers working on LLM evaluation, test-time compute, and reward modeling
- Coding-agent and software-engineering automation users
- Robotics and embodied-AI researchers
- +2 more
AI Tools by LLM-as-a-Verifier Team
(1)
LLM-as-a-Verifier
Python Framework for Agent Scoring
Discussions
No discussions yet
Be the first to start a discussion about LLM-as-a-Verifier Team
Latest News
Release 0.3.0: Terminal-Bench 4.0 and LLM-as-a-Verifier V3
TurboAgent update adds model-prefix routing, DeepSeek/OpenAI-compatible backends, provider checks, and opencode support documentation
Release v0.2.0 adds DeepSeek V4 Flash verification, prefix-cache optimization, token accounting, and Terminal-Bench 2.1 self-verification
TurboAgent publish commit introduces the Claude Code plugin/API-proxy extension
Products & Services
Open-source general-purpose verification framework installable with pip. Its select API performs best-of-N selection, compare performs pairwise fine-grained scoring, and track estimates progress over trajectory steps. It supports criteria decomposition, repeated evaluation, continuous rewards, multimodal images/video, and the O(Nk) Probabilistic Pivot Tournament.
A Claude Code plugin and OpenAI-compatible LLM API proxy that generates multiple candidate responses concurrently, verifies them with the Probabilistic Pivot Tournament, selects the best response, and exposes progress/candidate visualization.
Market Position
The project positions itself against coarse LLM-as-a-Judge systems and learned reward models by using the full scoring-token distribution, repeated evaluation, and decomposed criteria without additional training. The paper also compares or discusses frontier-model judges and robotics reward models including Robometer, TOPReward, and RoboReward; its claimed differentiation is a general-purpose, modality-agnostic verifier with cost-efficient O(Nk) selection, continuous feedback, and extensions for coding agents.
Leadership
Founders
Jacky Kwok
Stanford University researcher; corresponding author and project lead listed on the paper and website.
Shulu Li
University of California, Berkeley researcher.
Pranav Atreya
University of California, Berkeley researcher.
Yuejiang Liu
Stanford University researcher.
Yixing Jiang
Stanford University researcher.
Chelsea Finn
Stanford University researcher and faculty member in the project affiliation list.
Marco Pavone
Stanford University and NVIDIA Research researcher.
Ion Stoica
University of California, Berkeley researcher.
Azalia Mirhoseini
Stanford University researcher.
Founding Story
The team argues that generation is no longer the only important scaling axis: agents can often produce a correct solution among many attempts but cannot reliably identify it. The project was started to make verification itself scalable, replacing coarse discrete LLM judging with probabilistic, fine-grained scoring that reduces ties and can also provide dense progress and reinforcement-learning signals without additional verifier training.
Business Model
Revenue Model
The project is distributed as open-source Python packages (llm-verifier and TurboAgent) and is operated using the user's own model-provider/API credentials or an OpenAI-compatible logprob server. The public sources describe installation and API use but do not describe a paid commercial service.
Target Markets
- AI-agent developers and engineering teams
- Researchers working on LLM evaluation, test-time compute, and reward modeling
- Coding-agent and software-engineering automation users
- Robotics and embodied-AI researchers
- Medical-agent researchers
- Reinforcement-learning researchers
- Test-time scaling and best-of-N selection for coding agents
- Progress monitoring and safe deployment of long-horizon agents
- Robotics rollout evaluation and visual trajectory scoring
- Medical-agent evaluation
- Dense rewards for off-policy SAC and on-policy GRPO reinforcement learning
- Self-verification and cross-model verification of agent trajectories