Open source Python framework from Stanford CRFM for holistic, reproducible, and transparent evaluation of language and multimodal foundation models.
At a Glance
About HELM
Holistic Evaluation of Language Models (HELM) is an open source Python framework created by Stanford's Center for Research on Foundation Models (CRFM). It runs standardized benchmarks against models from various providers, computes metrics beyond accuracy, and presents results in a web UI and leaderboards. The README notes that HELM entered maintenance mode on June 1, 2026.
What It Is
HELM is a evaluation framework for foundation models, including large language models and multimodal models. It provides datasets and benchmarks in a standardized format (for example MMLU-Pro, GPQA, IFEval, WildBench) and a unified interface to models from providers such as OpenAI, Anthropic Claude, and Google Gemini. Metrics cover aspects such as efficiency, bias, and toxicity in addition to accuracy.
Workflow and Outputs
After installing the crfm-helm package from PyPI, users run helm-run with run entries (benchmark, subject, model), summarize results with helm-summarize, and launch helm-server to browse results locally in a browser. The web UI allows inspecting individual prompts and responses, and a web leaderboard compares results across models and benchmarks.
Official Leaderboards
The project maintains leaderboards built on the framework. Flagship ones are HELM Capabilities, HELM Safety, and VHELM for vision-language models. Others cover text-to-image models (HEIM), audio-language models, medicine (MedHELM), finance, long context, table reasoning (ToRR), and language-specific benchmarks such as Arabic, Thai, Chinese, and Southeast Asian languages. The framework can be used to reproduce the results published in the associated papers.
Current Status
The README states HELM entered maintenance mode on June 1, 2026, with details in a Maintenance Mode Policy. The latest listed release is v0.5.16, published 2026-04-30. It is licensed under Apache 2.0.
Community Discussions
Be the first to start a conversation about HELM
Share your experience with HELM, ask questions, or help others learn from your insights.
Pricing
Open Source
HELM is an open source Python framework released under the Apache License 2.0; install from PyPI and run locally.
- Apache License 2.0
- Datasets and benchmarks in a standardized format
- Models from various providers accessible through a unified interface
- Metrics beyond accuracy (efficiency, bias, toxicity)
- Web UI for inspecting prompts and responses
Capabilities
Key Features
- Standardized datasets and benchmarks
- Unified interface to models from multiple providers
- Metrics beyond accuracy such as efficiency, bias, and toxicity
- Web UI for inspecting individual prompts and responses
- Web leaderboards comparing models and benchmarks
- Multimodal evaluation (vision-language, audio-language, text-to-image)
- Reproduction of published leaderboard results
- CLI commands: helm-run, helm-summarize, helm-server