EvalPlus

EvalPlus is an open-source evaluation framework for LLMs on code tasks, published by the evalplus organization on GitHub with an accompanying benchmarks site and leaderboards. It extends the HumanEval and MBPP benchmarks with far more tests and adds an efficiency benchmark called EvalPerf. The latest documented release is v0.3.1, dated 2024-10-20.

What It Is

EvalPlus is a Python package and Docker image for measuring how well language models write code. It provides HumanEval+ (80x more tests than the original HumanEval) and MBPP+ (35x more tests than the original MBPP) for correctness. EvalPerf evaluates the efficiency of generated code using performance-exercising coding tasks and test inputs, based on Differential Performance Evaluation from a COLM'24 paper. The project is associated with NeurIPS 2023 and COLM 2024 papers.

How the Evaluation Workflow Works

Users install the package with pip and run commands such as evalplus.evaluate, choosing a model, a dataset (humaneval or mbpp), and an inference backend. evalplus.codegen generates samples, and evalplus.evalperf runs efficiency evaluation (Unix-like systems only, with perf enabled). Generated code can be executed inside Docker for safer execution. Results and generations are written to an evalplus_results directory. A single command can run generation, post-processing and evaluation. Base and chat models use different prompts, detected from the tokenizer chat template for the hf and vllm backends.

Backends and Model Access

Supported backends include Hugging Face transformers, vLLM, OpenAI-compatible servers, OpenAI, Anthropic, Google Gemini, Amazon Bedrock, Ollama, GPTQModel and Intel Gaudi (hf_gaudi). OpenAI-compatible endpoints can be used with services such as DeepSeek and Grok via a base URL.

Leaderboards and Wider Benchmarks

The EvalPlus site hosts a leaderboard comparing scores before and after the extended tests, and an EvalPerf leaderboard. The README says the datasets have been used by teams behind models such as Meta Llama 3.1, Qwen2.5-Coder and StarCoder2. The site also describes RepoQA, an effort on long-context code understanding.