PACE-Bench
A benchmark and evaluation toolkit that measures whether coding agents fully deliver enterprise PRDs in existing repositories.
At a Glance
The public checkout includes the evaluation toolkit and one anonymized example task, accessible without payment. No license file is declared; do not assume an open-source license or unrestricted reuse. Online construction and semantic judging require a separately configured model service and may incur provider costs. The full 43-task business repositories and patches are not distributed.
Engagement
Available On
Alternatives
Listed Nov 2026
About PACE-Bench
PACE-Bench (PRD-Anchored Coding-agent Evaluation) is a research benchmark and publicly available toolkit from Tianqi Liu and co-authors. It evaluates whether coding agents complete evolving product requirements in mature enterprise repositories, and it links unmet requirements to code and execution evidence. The paper suite covers 43 incremental enterprise tasks with 750 expert-audited Code Points.
What It Is
PACE-Bench is an evaluation framework for coding agents. Each task gives an agent a base repository and a final Product Requirement Document (PRD), and the agent's resulting patch is scored against frozen acceptance contracts called Code Points. Each Code Point receives a verdict of Covered (1), Partial (0.5) or Missing (0). Complete delivery is defined as a case score of at least 95 with zero Missing Code Points.
How the Evaluation Works
The method has four stages: intent-preserving PRD reconstruction, bidirectional Code Point synthesis, reference-grounded contract validation, and symptom–signal attribution. The scorer applies three evidence stages: contract checks for conclusive violations, evidence localization, and constrained semantic judgment. Outputs are cited point verdicts, a weighted case score, and a diagnosis linking unmet requirements to code evidence and execution signals. Attribution does not change the functional score.
Reported Results
Across 11 model–scaffold settings, the authors report a highest complete-delivery rate of 14.0%. They report that the full evaluator reaches 95.2% agreement with expert consensus. The project page also reports code-quality, execution-observability, stability and static-knowledge analyses.
Setup Path
The repository requires Python 3.11 or newer and has no runtime dependencies for input preparation and sandbox export. A configured model service is needed for online construction and semantic judging. A pace prepare command prepares a task directory or a PRD file. The public checkout includes one anonymized real task with an English PRD and 17 weighted Code Points; original business repositories and patches are not distributed. The framework does not require a particular coding-agent framework, and teams supply their own agent runner.
Community Discussions
Be the first to start a conversation about PACE-Bench
Share your experience with PACE-Bench, ask questions, or help others learn from your insights.
Pricing
Public research checkout
The public checkout includes the evaluation toolkit and one anonymized example task, accessible without payment. No license file is declared; do not assume an open-source license or unrestricted reuse. Online construction and semantic judging require a separately configured model service and may incur provider costs. The full 43-task business repositories and patches are not distributed.
- Evaluation toolkit
- One public example with 17 weighted Code Points
- Offline input preparation and sandbox export
Capabilities
Key Features
- Benchmark of 43 enterprise PRD tasks with 750 expert-audited Code Points
- Covered/Partial/Missing point verdicts with weighted case scoring
- Intent-preserving PRD reconstruction
- Bidirectional Code Point synthesis
- Three-stage evaluation: contract, localization, semantic judgment
- Symptom–signal attribution linking gaps to execution traces
- Code-quality and execution-observability reporting
- pace prepare CLI for task inputs and sandbox export
- Anonymized real example task with 17 weighted Code Points