# MTEB

> Python toolbox for benchmarking and evaluating embedding models and retrieval systems across tasks, languages, and modalities.

MTEB (Massive Text Embedding Benchmark) is an open-source Python package from the embeddings-benchmark organization for evaluating the quality of embeddings. It lets users pick a model, select tasks or benchmarks, and run evaluations from a script or the command line. The documentation describes more than 1000 tasks across more than 1000 languages, covering text and image tasks, with audio support also referenced.

## What It Is

MTEB is an evaluation toolkit for embedding models and retrieval systems. It started as a package for evaluating text embeddings, predominantly in English, and was later extended to broader language coverage and to multiple modalities, including image and audio. It is installed with pip or uv and released under the Apache 2.0 license.

## How an Evaluation Works

The workflow has three steps: load a model with `mteb.get_model`, choose tasks with `mteb.get_tasks` or a benchmark with `mteb.get_benchmark`, and call `mteb.evaluate`. If a model is not implemented in MTEB, it falls back to a SentenceTransformer. The same run is available from the CLI with `mteb run`, which takes a model, task names, and an output folder. Benchmark objects expose citation information and can export their task list as a LaTeX table.

## Ecosystem and Extensions

The docs include guides for loading existing results, caching embeddings, BM25 baselines, hybrid models, retrieval backends, two-stage reranking, and vLLM and OpenAI-compatible API wrappers. Task categories include retrieval, classification, clustering, semantic similarity, and pair classification. Contributors can add models, tasks, benchmarks, and submit results, and a leaderboard can be run locally or viewed online.

## Features
- Evaluate embedding models across 1000+ tasks
- Multilingual coverage across 1000+ languages
- Text, image, and audio modality support
- Python API and CLI (mteb run)
- Benchmark selection with citation and LaTeX task export
- Load and work with existing model results
- Result caching and embedding cache
- BM25 baselines, hybrid models, and two-stage reranking
- vLLM and OpenAI-compatible API wrappers
- Local leaderboard and contribution workflows for models, tasks, and benchmarks

## Integrations
Sentence Transformers, Hugging Face, vLLM, OpenAI-compatible APIs

## Platforms
API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
2.24.12

## Links
- Website: https://docs.mteb.org
- Documentation: https://docs.mteb.org
- Repository: https://github.com/embeddings-benchmark/mteb
- EveryDev.ai: https://www.everydev.ai/tools/mteb