Inspect Evals

Inspect Evals is a library of evaluations built using Inspect AI. It is maintained by Generality Labs, a London-based nonprofit, and was founded in 2024 with contributions from the UK AI Security Institute, Arcadia Impact, and the Vector Institute. The documentation site catalogues 184 evaluations, of which 129 live in the repository and 55 are external listings.

What It Is

Inspect Evals is a collection of ready-to-run benchmarks and safety evaluations for language models and AI agents. Each evaluation is an Inspect AI task that you run against a model from your chosen provider. Coverage in the catalogue spans several areas: coding, math and reasoning, cybersecurity and capture-the-flag challenges, agent safety and prompt injection, alignment and deception, bias, biology and chemistry knowledge, and multimodal understanding.

How It Works

You install the package from PyPI with pip install inspect-evals, or sync a checked-out repository with uv sync. You then run tasks with the Inspect CLI, for example inspect eval inspect_evals/arc_easy --model openai/gpt-5-nano, or run several together with inspect eval-set. Tasks can also be imported as Python objects and run from code. Logs from runs can be examined with inspect view. Inspect supports many model providers, including OpenAI, Anthropic, Google, Mistral, AWS Bedrock, Hugging Face, vLLM, and Ollama.

Register and Contribution

The Inspect Evals Register lets authors add a listing that points to their own implementation and documentation. The register is where the external evaluations in the catalogue come from. The repository currently accepts pull requests only from pre-approved contributors, and issues should be raised with .eval logs that demonstrate the problem.

Setup Notes

Python 3.11 or 3.12 is recommended. The README suggests at least 35 GB of free disk space, and about 100 GB for evals that pull Docker images. Most evals need around 0.5 GB of RAM, but some Docker-based evals need up to 32 GB. Cache location can be changed with the INSPECT_EVALS_CACHE_DIR variable.