# HELM

> Open source Python framework from Stanford CRFM for holistic, reproducible, and transparent evaluation of language and multimodal foundation models.

Holistic Evaluation of Language Models (HELM) is an open source Python framework created by Stanford's Center for Research on Foundation Models (CRFM). It runs standardized benchmarks against models from various providers, computes metrics beyond accuracy, and presents results in a web UI and leaderboards. The README notes that HELM entered maintenance mode on June 1, 2026.

## What It Is

HELM is a evaluation framework for foundation models, including large language models and multimodal models. It provides datasets and benchmarks in a standardized format (for example MMLU-Pro, GPQA, IFEval, WildBench) and a unified interface to models from providers such as OpenAI, Anthropic Claude, and Google Gemini. Metrics cover aspects such as efficiency, bias, and toxicity in addition to accuracy.

## Workflow and Outputs

After installing the crfm-helm package from PyPI, users run helm-run with run entries (benchmark, subject, model), summarize results with helm-summarize, and launch helm-server to browse results locally in a browser. The web UI allows inspecting individual prompts and responses, and a web leaderboard compares results across models and benchmarks.

## Official Leaderboards

The project maintains leaderboards built on the framework. Flagship ones are HELM Capabilities, HELM Safety, and VHELM for vision-language models. Others cover text-to-image models (HEIM), audio-language models, medicine (MedHELM), finance, long context, table reasoning (ToRR), and language-specific benchmarks such as Arabic, Thai, Chinese, and Southeast Asian languages. The framework can be used to reproduce the results published in the associated papers.

## Current Status

The README states HELM entered maintenance mode on June 1, 2026, with details in a Maintenance Mode Policy. The latest listed release is v0.5.16, published 2026-04-30. It is licensed under Apache 2.0.

## Features
- Standardized datasets and benchmarks
- Unified interface to models from multiple providers
- Metrics beyond accuracy such as efficiency, bias, and toxicity
- Web UI for inspecting individual prompts and responses
- Web leaderboards comparing models and benchmarks
- Multimodal evaluation (vision-language, audio-language, text-to-image)
- Reproduction of published leaderboard results
- CLI commands: helm-run, helm-summarize, helm-server

## Integrations
OpenAI, Anthropic Claude, Google Gemini, PyPI

## Platforms
CLI, WEB, DEVELOPER_SDK

## Pricing
Open Source

## Version
v0.5.16

## Links
- Website: https://crfm.stanford.edu/helm
- Documentation: https://crfm-helm.readthedocs.io/en/latest/
- Repository: https://github.com/stanford-crfm/helm
- EveryDev.ai: https://www.everydev.ai/tools/helm