evalplus
EvalPlus is an open-source research project and evaluation framework for LLM-generated code. It builds rigorous, high-quality evaluators and benchmarks for code correctness, code efficiency, and long-context code understanding.
At a Glance
- Academic researchers in machine learning, program synthesis, and software engineering
- LLM and code-model developers
- AI labs and model-evaluation teams
- Open-source model communities
- +1 more
AI Tools by evalplus
(1)
EvalPlus
Open Source LLM Code Benchmark
Discussions
No discussions yet
Be the first to start a discussion about evalplus
Latest News
EvalPlus repository adds Intel Gaudi accelerator support, including a Hugging Face Gaudi backend and vLLM example.
University of Illinois reports that Jiawei Liu received the 2025 Illinois Innovation Award and notes EvalPlus had been downloaded 750,000 times.
RepoQA repository publishes its latest documented update, refining the Hugging Face backend and stop-string handling.
EvalPlus v0.3.1 release adds EvalPerf, end-to-end evaluation workflow, and Google Gemini/Anthropic backend support.
Products & Services
Open-source Python tooling for code generation, sanitization/post-processing, syntax checking, sandboxed evaluation, and benchmarking. Its documented commands include evalplus.codegen, evalplus.evaluate, evalplus.sanitize, and evalplus.syncheck.
Enhanced HumanEval benchmark with approximately 80x more tests, designed to expose incorrect LLM-generated solutions that pass the original limited tests.
Enhanced Mostly Basic Python Problems benchmark; the v0.2.0 update retained 378 tasks after removing broken tasks and improving ground-truth solutions.
Code-efficiency evaluation dataset and leaderboard using performance-exercising coding tasks and challenging inputs; the site describes 118 tasks and differential performance evaluation.
Market Position
EvalPlus differentiates from baseline HumanEval and MBPP by substantially expanding and repairing their tests, making correctness claims more rigorous. EvalPerf extends the positioning beyond correctness into code efficiency, while RepoQA targets long-context code understanding; the RepoQA paper places it alongside related code benchmarks such as RepoBench, CrossCodeEval, SWE-bench, LongBench, and Ruler but describes RepoQA as the first benchmark focused on long-context code understanding.
Leadership
Founders
Jiawei Liu
University of Illinois Urbana-Champaign computer-science researcher and Ph.D. student at the time of the EvalPlus work; research focuses on reliable code-generating models, automatic testing, and dependable ML systems.
Chunqiu Steven Xia
Co-author of the EvalPlus NeurIPS 2023 paper; the paper lists him as affiliated with Nanjing University.
Yuyao Wang
Co-author of the EvalPlus NeurIPS 2023 paper and software-engineering/ML research collaborator with the project team.
Lingming Zhang
University of Illinois Urbana-Champaign professor/researcher and co-author of the EvalPlus NeurIPS 2023 paper; leads research in software engineering and program analysis.
Founding Story
EvalPlus began from the observation that standard code-generation benchmarks such as HumanEval had too few or insufficiently strong tests to establish whether LLM-generated programs were truly correct. The team proposed an automated, mutation- and LLM-assisted test-generation framework and released enhanced benchmarks and tools so the research community could evaluate functional correctness more rigorously.
Business Model
Revenue Model
The project is distributed as open-source software and datasets under Apache-2.0 repositories and through PyPI/Docker/Hugging Face; public sources describe no paid plans, hosted API, or subscription business.
Pricing Tiers
Installable from PyPI or the GitHub repository; Docker images and benchmark resources are also publicly available.
Target Markets
- Academic researchers in machine learning, program synthesis, and software engineering
- LLM and code-model developers
- AI labs and model-evaluation teams
- Open-source model communities
- Developers evaluating code-generation assistants and repository-level coding systems
- Benchmarking LLM code-generation models for functional correctness
- Detecting fragile or test-insufficient generated programs
- Comparing model code efficiency and runtime performance
- Evaluating long-context code search and repository understanding
- Running reproducible local or Docker-isolated evaluations
- Using enhanced datasets in academic and industrial LLM-for-code research
- Google, Meta, and Amazon are cited by the University of Illinois as industry leaders whose research communities have adopted EvalPlus and related open-source code-generation evaluation tools.