Massive Text Embedding Benchmark
Massive Text Embedding Benchmark (MTEB) is an open-source, community-driven research project and Python framework for evaluating embedding and retrieval systems across languages, modalities, and task types. It provides reproducible evaluation code, datasets/task definitions, public results, and a leaderboard so researchers and practitioners can compare models for their use cases. Sources: https://docs.mteb.org/overview/ and https://github.com/embeddings-benchmark/mteb
At a Glance
- NLP and information-retrieval researchers
- Embedding-model and foundation-model developers
- Search, recommendation, and RAG engineers
- Organizations evaluating multilingual, domain-specific, or multimodal models
- +1 more
AI Tools by Massive Text Embedding Benchmark
(1)
MTEB
Embedding Model Benchmark Toolkit
Discussions
No discussions yet
Be the first to start a discussion about Massive Text Embedding Benchmark
Latest News
MTEB announced a faster, feature-rich leaderboard redesign with improved filtering, model comparison, transparency, and scalability.
MTEB v2 launched with multimodal evaluation, non-embedding retrieval support, a consistent interface, improved caching, search interfaces, richer statistics, and new documentation.
MMTEB was submitted as a major multilingual, community-driven expansion of MTEB, covering 500+ tasks across 250+ languages.
MTEB documentation highlighted MIEB, the Massive Image Embedding Benchmark, as an image/image-text extension supported by the broader v2 multimodal architecture.
Products & Services
Open-source Python package for selecting models and tasks, running evaluations, loading and comparing results, caching embeddings/results, and submitting model results. It covers classification, clustering, semantic similarity, pair classification, retrieval, reranking, and related task types. Source: https://github.com/embeddings-benchmark/mteb
Public interactive leaderboard for comparing embedding and retrieval models across benchmarks and tasks; the 2026 version adds filtering by domain, language, modality, and task, task/data inspection, model comparison, and performance-versus-runtime analysis. Sources: https://huggingface.co/spaces/mteb/leaderboard and https://docs.mteb.org/blog/mteb-leaderboard-from-a-slow-demo-to-feature-rich-leaderboard/
Community-driven multilingual expansion covering 500+ quality-controlled evaluation tasks across 250+ languages, including instruction following, long-document retrieval, and code retrieval. Source: https://arxiv.org/abs/2502.13595
Image and image-text extension of MTEB; MTEB documentation identifies it as one of the major extensions enabled by the project and MTEB v2 supports multimodal evaluation. Sources: https://docs.mteb.org/blog/introducing-mteb-v2-evaluation-of-embedding-and-retrieval-systems-for-more-than-just-text/ and https://arxiv.org/abs/2504.10471
Market Position
MTEB positions itself as a broad, open, reproducible evaluation layer rather than a model vendor: the original benchmark expanded beyond narrow semantic-textual-similarity and retrieval evaluations to 8 tasks and 58 datasets, while later work expanded language, domain, and modality coverage. Relative to narrower suites such as BEIR or STS-only evaluations, MTEB offers a unified task/model interface, community extensibility, public leaderboard, and increasingly multilingual and multimodal coverage. Sources: https://arxiv.org/abs/2210.07316 and https://docs.mteb.org/blog/introducing-mteb-v2-evaluation-of-embedding-and-retrieval-systems-for-more-than-just-text/
Leadership
Founders
Niklas Muennighoff
Lead author of the original MTEB paper and creator/contributor of the MTEB codebase; his personal site identifies MTEB as one of his major research projects and describes him as an AI/ML researcher. Sources: https://arxiv.org/abs/2210.07316 and https://muennighoff.com/
Nouamane Tazi
Co-author of the original MTEB research paper and contributor to the initial benchmark project. Source: https://aclanthology.org/2023.eacl-main.148/
Loïc Magne
Co-author of the original MTEB research paper and contributor to the initial benchmark project. Source: https://aclanthology.org/2023.eacl-main.148/
Nils Reimers
Co-author of the original MTEB research paper and contributor to the initial benchmark project. Source: https://aclanthology.org/2023.eacl-main.148/
Founding Story
MTEB was started because text embeddings were commonly evaluated on a small number of datasets from a single task, making it difficult to tell whether a model generalized to clustering, reranking, retrieval, classification, and other applications. The founders introduced a broad benchmark in 2022 to make progress easier to track, initially covering 8 tasks, 58 datasets, and 112 languages, with open-source code and a public leaderboard. Source: https://arxiv.org/abs/2210.07316
Business Model
Revenue Model
Open-source research infrastructure: the MTEB code is Apache-2.0 licensed, and the benchmark, documentation, and public leaderboard are provided openly. The inspected sources describe no paid product, subscription, API-usage charge, or commercial revenue model. Source: https://github.com/embeddings-benchmark/mteb/blob/main/LICENSE
Target Markets
- NLP and information-retrieval researchers
- Embedding-model and foundation-model developers
- Search, recommendation, and RAG engineers
- Organizations evaluating multilingual, domain-specific, or multimodal models
- Open-source AI and academic benchmarking communities
- Selecting embedding models for semantic search and retrieval-augmented generation
- Evaluating models for classification, clustering, semantic similarity, reranking, and bitext mining
- Comparing multilingual and domain-specific models
- Evaluating multimodal embedding and retrieval systems
- Reproducible academic research and benchmarking
- Submitting new models, datasets, benchmarks, and results to a public leaderboard