LM Evaluation Harness

LM Evaluation Harness is an open-source Python framework from EleutherAI, a non-profit AI research lab, for testing generative language models on a large set of evaluation tasks. It is run from a command line or Python API and reports benchmark results for models loaded locally or reached through hosted APIs. The latest release listed on GitHub is v0.4.13.

What It Is

The harness provides a unified way to evaluate language models against more than 60 standard academic benchmarks, with hundreds of subtasks and variants. Users choose a model backend, pick tasks, and run the evaluation; results can be saved, cached, and logged with per-sample outputs. EleutherAI describes it as the framework behind the Open LLM Leaderboard and its evaluation framework for open models.

Model Backends and Inference

Backends are installed as optional extras, so the base package stays light. Documented backends include Hugging Face transformers (with GPTQ quantization and PEFT adapters), vLLM, SGLang, NVIDIA NeMo, Megatron-LM, OpenVINO, IPEX, Habana, AWS Neuron, ONNX Runtime and ONNX Runtime GenAI, Windows ML, and llama.cpp servers. Commercial and OpenAI-compatible APIs such as OpenAI, Anthropic, TextSynth, and LiteLLM are also supported. Multi-GPU evaluation is available through accelerate data parallelism, model sharding, tensor parallelism, and vLLM parallel settings. A steered model type applies steering vectors to Hugging Face models.

Tasks, Prompts, and Results

Tasks are defined through YAML configuration, with Jinja2 prompt templates and support for custom prompts and metrics. Plugins let you register backends, filters, and metrics from your own package. Evaluations use publicly available prompts, which the README says supports reproducibility and comparability between papers. Results can be written to disk, pushed to the Hugging Face Hub, and visualized with Weights & Biases or Zeno.

Setup Path

Install from the GitHub repository with pip, then add a backend extra such as hf, vllm, or api. The CLI offers run, ls, and validate subcommands, and YAML config files can be passed with --config. Support is through GitHub issues and the EleutherAI Discord.