BigCode Evaluation Harness

BigCode Evaluation Harness is an open-source Python framework from the bigcode-project organization for evaluating autoregressive code generation models. It is inspired by EleutherAI's lm-evaluation-harness and runs Hugging Face hub models against code benchmarks, generating solutions and optionally executing them against unit tests.

What It Is

The harness is a command-line evaluation framework. Users point it at a model and one or more benchmark tasks, and it generates code or text completions, post-processes them, and scores them. Any autoregressive model on the Hugging Face hub can be used, though the README recommends models trained on code such as SantaCoder, InCoder and CodeGen.

Supported Tasks

The README lists Python tasks with unit tests (HumanEval, HumanEval+, InstructHumanEval, APPS, MBPP, MBPP+, DS-1000) in completion and fill-in-the-middle modes. It also covers HumanEvalPack, MultiPL-E (HumanEval translated into 18 languages), Recode for robustness, PAL for GSM8K and GSM-HARD math problems, CodeXGLUE code-to-text, CoNaLa, Concode, and three classification tasks. SantaCoder-FIM and Mercury, which measures computational efficiency, are also included.

Workflow and Execution Safety

Generation and evaluation can run together, or separately with generation-only and evaluation-only modes. This lets users generate on GPUs and execute the code on CPU machines or in Docker containers. Code execution is off by default and must be enabled with a flag. Multi-GPU generation uses accelerate, and models can be loaded at reduced precision or in 8-bit or 4-bit. Dockerfiles are provided, including one with the extra dependencies MultiPL-E needs. Contributors can add new tasks by following the guide in the docs.