# PACE-Bench

> A benchmark and evaluation toolkit that measures whether coding agents fully deliver enterprise PRDs in existing repositories.

PACE-Bench (PRD-Anchored Coding-agent Evaluation) is a research benchmark and publicly available toolkit from Tianqi Liu and co-authors. It evaluates whether coding agents complete evolving product requirements in mature enterprise repositories, and it links unmet requirements to code and execution evidence. The paper suite covers 43 incremental enterprise tasks with 750 expert-audited Code Points.

## What It Is

PACE-Bench is an evaluation framework for coding agents. Each task gives an agent a base repository and a final Product Requirement Document (PRD), and the agent's resulting patch is scored against frozen acceptance contracts called Code Points. Each Code Point receives a verdict of Covered (1), Partial (0.5) or Missing (0). Complete delivery is defined as a case score of at least 95 with zero Missing Code Points.

## How the Evaluation Works

The method has four stages: intent-preserving PRD reconstruction, bidirectional Code Point synthesis, reference-grounded contract validation, and symptom–signal attribution. The scorer applies three evidence stages: contract checks for conclusive violations, evidence localization, and constrained semantic judgment. Outputs are cited point verdicts, a weighted case score, and a diagnosis linking unmet requirements to code evidence and execution signals. Attribution does not change the functional score.

## Reported Results

Across 11 model–scaffold settings, the authors report a highest complete-delivery rate of 14.0%. They report that the full evaluator reaches 95.2% agreement with expert consensus. The project page also reports code-quality, execution-observability, stability and static-knowledge analyses.

## Setup Path

The repository requires Python 3.11 or newer and has no runtime dependencies for input preparation and sandbox export. A configured model service is needed for online construction and semantic judging. A `pace prepare` command prepares a task directory or a PRD file. The public checkout includes one anonymized real task with an English PRD and 17 weighted Code Points; original business repositories and patches are not distributed. The framework does not require a particular coding-agent framework, and teams supply their own agent runner.

## Features
- Benchmark of 43 enterprise PRD tasks with 750 expert-audited Code Points
- Covered/Partial/Missing point verdicts with weighted case scoring
- Intent-preserving PRD reconstruction
- Bidirectional Code Point synthesis
- Three-stage evaluation: contract, localization, semantic judgment
- Symptom–signal attribution linking gaps to execution traces
- Code-quality and execution-observability reporting
- pace prepare CLI for task inputs and sandbox export
- Anonymized real example task with 17 weighted Code Points

## Platforms
DEVELOPER_SDK, CLI

## Pricing
Free

## Links
- Website: https://tianqiliu1.github.io/PACE-Bench/
- Documentation: https://github.com/TianqiLiu1/PACE-Bench/blob/main/docs/GETTING_STARTED.md
- Repository: https://github.com/TianqiLiu1/PACE-Bench
- EveryDev.ai: https://www.everydev.ai/tools/pace-bench