# OpenCompass

> Open-source platform for evaluating large language and multimodal models across 100+ datasets.

OpenCompass is an open-source LLM evaluation platform from the open-compass organization. It covers dataset and model configuration, inference, evaluation, and result summarization. The README lists support for models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek and others, and integrates over 100 datasets. The latest listed release is 0.5.4.

## What It Is

OpenCompass is a Python framework for benchmarking foundation models. Users configure models and datasets, run inference, score the outputs, and summarize the results. The datasets span knowledge, reasoning, coding, science, language, long context, and safety. Through VLMEvalKit integration, text-only and vision-language datasets can be evaluated in one workflow.

## How Evaluation Works

It supports local open-source models and API models, with inference backends including HuggingFace, LMDeploy, and vLLM. Evaluation modes include zero-shot, few-shot, chain-of-thought, rule-based, and LLM-as-judge. Task partitioning, concurrent execution, and distributed execution handle workloads of different scales. Users can plug in new models, datasets, evaluators, inference backends, and task schedulers.

## Recent Updates

The README's What's New list includes VLMEvalKit integration for multimodal datasets (2026.08.25). It also lists support for the OpenAI Responses API and LiteLLM AI Gateway (2026.07.28), multi-round inference with the Multi-IF dataset, and repeat-analysis tooling for looping outputs. Other entries cover concurrent inference with evaluation watching and RawPromptTemplate, which passes original benchmark prompts to models without formatting changes.

## Related Ecosystem

CompassHub provides benchmark navigation, and CompassRank presents model leaderboards. The OpenCompass website also hosts an Arena where users can vote on models.

## Features
- Evaluation across 100+ datasets covering knowledge, reasoning, coding, science, language, long context, and safety
- Support for open-source and API models
- Inference backends: HuggingFace, LMDeploy, vLLM
- Zero-shot, few-shot, chain-of-thought, rule-based, and LLM-as-judge evaluation
- Task partitioning, concurrent and distributed execution
- VLMEvalKit integration for multimodal evaluation
- CascadeEvaluator, GenericLLMEvaluator, and MATHVerifyEvaluator
- Repeat analysis tool for looping outputs
- Extensible models, datasets, evaluators, and schedulers

## Integrations
OpenAI, Anthropic Claude, Google Gemini, ByteDance Doubao, xAI Grok, HuggingFace, LMDeploy, vLLM, VLMEvalKit, LiteLLM, DeepSeek, Qwen

## Platforms
API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
0.5.4

## Links
- Website: https://opencompass.org.cn/
- Documentation: https://opencompass.readthedocs.io/en/latest/
- Repository: https://github.com/open-compass/opencompass
- EveryDev.ai: https://www.everydev.ai/tools/opencompass