text2vec

text2vec is an open-source Python toolkit by XuMing (shibing624) that turns text (words, sentences, paragraphs) into vector matrices and computes semantic similarity. It implements Word2Vec, Sentence-BERT, CoSENT and BGE representation models and compares them on text semantic matching tasks. The latest PyPI release is 1.3.8 (Nov 29, 2025), distributed under the Apache 2.0 license.

What It Is

text2vec is a text embedding library installed with pip. A SentenceModel loads a pretrained model (the default is shibing624/text2vec-base-chinese) and encodes sentences into numpy arrays, for example 768-dimensional embeddings. A Word2Vec model based on a lightweight Tencent AI Lab Chinese word-vector file is also provided, with sentence vectors computed by averaging word vectors.

Workflow and Outputs

• Compute embeddings in Python, or in batch through the text2vec command-line tool, which reads a one-sentence-per-line file and writes a CSV of embeddings. Multi-GPU and multi-CPU inference is supported.
• Score sentence similarity with cosine similarity in the range [-1, 1].
• Run semantic search that returns the top-k most similar sentences from a corpus for a query.
• Models can also be loaded through HuggingFace Transformers or sentence-transformers.

Training and Models

The repository includes training scripts for supervised and unsupervised CoSENT and SBERT models, BERT-Match, and BGE fine-tuning with (query, positive, negative) triples. Training data can be your own sentence pairs labeled 0/1 or on a 0-5 similarity scale. Released models cover Chinese, multilingual and BGE-based variants, with evaluation tables on ATEC, BQ, LCQMC, PAWSX, STS-B and SOHU sets. Embedding dimensionality reduction and distillation are supported by reusing sentence-transformers methods.

Deployment

The README describes serving models as a gRPC/HTTP service through Jina or as a native HTTP service with FastAPI. The project also lists released Chinese semantic matching datasets on HuggingFace.