OpenCompass
Open-source platform for evaluating large language and multimodal models across 100+ datasets.
At a Glance
About OpenCompass
OpenCompass is an open-source LLM evaluation platform from the open-compass organization. It covers dataset and model configuration, inference, evaluation, and result summarization. The README lists support for models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek and others, and integrates over 100 datasets. The latest listed release is 0.5.4.
What It Is
OpenCompass is a Python framework for benchmarking foundation models. Users configure models and datasets, run inference, score the outputs, and summarize the results. The datasets span knowledge, reasoning, coding, science, language, long context, and safety. Through VLMEvalKit integration, text-only and vision-language datasets can be evaluated in one workflow.
How Evaluation Works
It supports local open-source models and API models, with inference backends including HuggingFace, LMDeploy, and vLLM. Evaluation modes include zero-shot, few-shot, chain-of-thought, rule-based, and LLM-as-judge. Task partitioning, concurrent execution, and distributed execution handle workloads of different scales. Users can plug in new models, datasets, evaluators, inference backends, and task schedulers.
Recent Updates
The README's What's New list includes VLMEvalKit integration for multimodal datasets (2026.08.25). It also lists support for the OpenAI Responses API and LiteLLM AI Gateway (2026.07.28), multi-round inference with the Multi-IF dataset, and repeat-analysis tooling for looping outputs. Other entries cover concurrent inference with evaluation watching and RawPromptTemplate, which passes original benchmark prompts to models without formatting changes.
Related Ecosystem
CompassHub provides benchmark navigation, and CompassRank presents model leaderboards. The OpenCompass website also hosts an Arena where users can vote on models.
Community Discussions
Be the first to start a conversation about OpenCompass
Share your experience with OpenCompass, ask questions, or help others learn from your insights.
Pricing
Open Source
OpenCompass core evaluation platform released under the Apache License 2.0; free to use, modify and distribute under license terms.
- Apache License 2.0
- Supports models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek and more
- 100+ datasets covering knowledge, reasoning, coding, science, language, long-context and safety
- Inference backends: HuggingFace, LMDeploy, vLLM
- Zero-shot, few-shot, chain-of-thought, rule-based and LLM-as-judge evaluation
Capabilities
Key Features
- Evaluation across 100+ datasets covering knowledge, reasoning, coding, science, language, long context, and safety
- Support for open-source and API models
- Inference backends: HuggingFace, LMDeploy, vLLM
- Zero-shot, few-shot, chain-of-thought, rule-based, and LLM-as-judge evaluation
- Task partitioning, concurrent and distributed execution
- VLMEvalKit integration for multimodal evaluation
- CascadeEvaluator, GenericLLMEvaluator, and MATHVerifyEvaluator
- Repeat analysis tool for looping outputs
- Extensible models, datasets, evaluators, and schedulers