Lighteval

Lighteval is an open-source Python toolkit from Hugging Face's Leaderboard and Evals Team for evaluating large language models. It runs models served remotely or already loaded in memory against a library of benchmark tasks, and saves detailed sample-by-sample results. The latest release listed on GitHub is v0.13.0, published 2025-11-24.

What It Is

Lighteval is an LLM evaluation framework installed with pip. You choose a model, a backend, and one or more tasks, then run the evaluation from the command line or the Python API. Results can be inspected per sample to debug model behavior and compare models.

Backends and Workflow

The documentation lists entry points for several backends: inspect-ai (the preferred route), Hugging Face Accelerate on CPU or GPUs, Nanotron for distributed settings, vLLM, SGLang, Hugging Face Inference Endpoints, local Text Generation Inference, LiteLLM, Hugging Face inference providers, and custom models. A single command such as lighteval eval with a model name and a task like gpqa:diamond starts a run. The Python API exposes an EvaluationTracker and a Pipeline for models already loaded in memory.

Tasks and Customization

The README describes 1000+ evaluation tasks spanning knowledge, math and code, chat model evaluation, multilingual evaluation, and core language understanding. Examples include MMLU, GPQA, GSM8K, AIME, IFEval, MT-Bench, and Flores200. A Tasks Explorer lets you browse tasks by language and tag. Users can add custom tasks, custom metrics, and custom models. Results can be saved to the Hugging Face Hub, S3, or locally.

Setup Notes

The README states that Lighteval is untested on Windows and not supported there, while it should work on Mac and Linux. Pushing results to the Hub requires logging in with a Hugging Face access token.