MTEB

MTEB (Massive Text Embedding Benchmark) is an open-source Python package from the embeddings-benchmark organization for evaluating the quality of embeddings. It lets users pick a model, select tasks or benchmarks, and run evaluations from a script or the command line. The documentation describes more than 1000 tasks across more than 1000 languages, covering text and image tasks, with audio support also referenced.

What It Is

MTEB is an evaluation toolkit for embedding models and retrieval systems. It started as a package for evaluating text embeddings, predominantly in English, and was later extended to broader language coverage and to multiple modalities, including image and audio. It is installed with pip or uv and released under the Apache 2.0 license.

How an Evaluation Works

The workflow has three steps: load a model with mteb.get_model, choose tasks with mteb.get_tasks or a benchmark with mteb.get_benchmark, and call mteb.evaluate. If a model is not implemented in MTEB, it falls back to a SentenceTransformer. The same run is available from the CLI with mteb run, which takes a model, task names, and an output folder. Benchmark objects expose citation information and can export their task list as a LaTeX table.

Ecosystem and Extensions

The docs include guides for loading existing results, caching embeddings, BM25 baselines, hybrid models, retrieval backends, two-stage reranking, and vLLM and OpenAI-compatible API wrappers. Task categories include retrieval, classification, clustering, semantic similarity, and pair classification. Contributors can add models, tasks, benchmarks, and submit results, and a leaderboard can be run locally or viewed online.