# BigCode Evaluation Harness

> A framework for evaluating autoregressive code generation language models on a range of code benchmarks.

BigCode Evaluation Harness is an open-source Python framework from the bigcode-project organization for evaluating autoregressive code generation models. It is inspired by EleutherAI's lm-evaluation-harness and runs Hugging Face hub models against code benchmarks, generating solutions and optionally executing them against unit tests.

## What It Is

The harness is a command-line evaluation framework. Users point it at a model and one or more benchmark tasks, and it generates code or text completions, post-processes them, and scores them. Any autoregressive model on the Hugging Face hub can be used, though the README recommends models trained on code such as SantaCoder, InCoder and CodeGen.

## Supported Tasks

The README lists Python tasks with unit tests (HumanEval, HumanEval+, InstructHumanEval, APPS, MBPP, MBPP+, DS-1000) in completion and fill-in-the-middle modes. It also covers HumanEvalPack, MultiPL-E (HumanEval translated into 18 languages), Recode for robustness, PAL for GSM8K and GSM-HARD math problems, CodeXGLUE code-to-text, CoNaLa, Concode, and three classification tasks. SantaCoder-FIM and Mercury, which measures computational efficiency, are also included.

## Workflow and Execution Safety

Generation and evaluation can run together, or separately with generation-only and evaluation-only modes. This lets users generate on GPUs and execute the code on CPU machines or in Docker containers. Code execution is off by default and must be enabled with a flag. Multi-GPU generation uses accelerate, and models can be loaded at reduced precision or in 8-bit or 4-bit. Dockerfiles are provided, including one with the extra dependencies MultiPL-E needs. Contributors can add new tasks by following the guide in the docs.

## Features
- Evaluates any autoregressive Hugging Face hub model
- Multi-GPU text generation with accelerate
- Docker images for sandboxed code execution
- Generation-only and evaluation-only modes
- Python benchmarks with unit tests (HumanEval, MBPP, APPS, DS-1000)
- MultiPL-E multilingual evaluation
- Fill-in-the-middle evaluation tasks
- Code-to-text, classification and robustness tasks
- 8-bit and 4-bit model loading
- Extensible with new task implementations

## Integrations
Hugging Face Hub, accelerate, Docker, PyTorch, bitsandbytes

## Platforms
CLI, LINUX, MACOS

## Pricing
Open Source

## Version
v0.1.0

## Links
- Website: https://github.com/bigcode-project/bigcode-evaluation-harness
- Documentation: https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/docs/README.md
- Repository: https://github.com/bigcode-project/bigcode-evaluation-harness
- EveryDev.ai: https://www.everydev.ai/tools/bigcode-evaluation-harness