BM25S

BM25S (BM25-Sparse) is an open-source Python library by Xing Han Lù that ranks documents against a query using BM25 algorithms. It is built on Numpy and Scipy, with optional dependencies for stemming and Numba acceleration, and runs without Java or a separate search server.

What It Is

BM25S is a lexical search and retrieval library. You tokenize a corpus, index it, and retrieve the top-k documents and scores for one or more queries. The project says a sparse eager scoring strategy, which precomputes scores for document tokens, gives query speeds comparable to or exceeding Elasticsearch in its benchmarks. Indices and corpora can be saved to disk and reloaded later.

Workflow and Variants

The built-in tokenizer supports stopwords, stemmers, and custom splitters, and a Tokenizer class can save and load its vocabulary. The library supports several BM25 variants: Robertson, ATIRE, BM25L, BM25+ and Lucene (the default). Parameters such as k1, b, and delta are configurable. Corpus entries can be plain strings or dictionaries, and retrieval is position-based.

Interfaces and Integrations

Beyond the Python API, a high-level module indexes local CSV, JSON, JSONL, or TXT files. A bm25 command-line tool handles indexing and searching, including a user directory for saved indices. A built-in MCP server exposes retrieve and get_info tools to LLM agents such as Claude Desktop. Indices can be pushed to and loaded from the Hugging Face Hub.

Memory and Scope

Memory-mapped loading lets an index stay on disk and load on demand, which the project's benchmarks show reduces RAM use. The homepage states that BM25S is not a replacement for Elasticsearch, but a lightweight alternative for BM25 ranking, useful in research and prototyping where BM25 parameters and tokenization need precise control.