Model2Vec

Model2Vec is an open-source Python package from MinishLab that turns a sentence transformer into a compact static embedding model. The project states it reduces model size by up to 50x and makes inference up to 500x faster, with a small drop in performance. It ships with pre-trained Potion models on the HuggingFace Hub and is released under the MIT license.

What It Is

Model2Vec is a static embedding toolkit. Instead of running a full transformer at inference time, it computes one fixed vector per token and produces sentence embeddings by averaging those token vectors. Inference is therefore a lookup into an embedding matrix, which the docs say gives high throughput on CPU. Embeddings can be used for text classification, retrieval, clustering, or RAG systems.

How Distillation Works

Distillation forward-passes the vocabulary of a chosen base model through it to create initial token embeddings. It then applies PCA for dimensionality reduction and Smooth Inverse Frequency token weighting, approximating token probabilities from vocabulary rank via Zipf's law, so no corpus is needed. Optional steps include quantization to 16-bit floats or 8-bit integers, vocabulary clustering with k-means, and pre-training with Tokenlearn, which is how the Potion models are created. The README says a model can be distilled in about 30 seconds on a CPU without a dataset.

Usage and Models

The base package installs with pip and depends mainly on numpy; extras exist for distillation and training. Users load models with StaticModel.from_pretrained and call encode or encode_as_sequence. Distilled or pre-trained models can be fine-tuned for classification, similarity, regression, or pair similarity and run without torch. Listed models include potion-base-32M, potion-base-8M, potion-retrieval-32M, potion-multilingual-128M, and potion-code-16M-v2.

Ecosystem and Integrations

Model2Vec integrates with Sentence Transformers and LangChain and works with the HuggingFace Hub for sharing and loading models. Minish also publishes related packages such as Tokenlearn, Model2Vec-rs for Rust inference, SemHash, Vicinity, and Semble.