# text2vec

> Python toolkit that converts words, sentences and paragraphs into vectors and computes text similarity using Word2Vec, SBERT, CoSENT and BGE models.

text2vec is an open-source Python toolkit by XuMing (shibing624) that turns text (words, sentences, paragraphs) into vector matrices and computes semantic similarity. It implements Word2Vec, Sentence-BERT, CoSENT and BGE representation models and compares them on text semantic matching tasks. The latest PyPI release is 1.3.8 (Nov 29, 2025), distributed under the Apache 2.0 license.

## What It Is

text2vec is a text embedding library installed with pip. A SentenceModel loads a pretrained model (the default is shibing624/text2vec-base-chinese) and encodes sentences into numpy arrays, for example 768-dimensional embeddings. A Word2Vec model based on a lightweight Tencent AI Lab Chinese word-vector file is also provided, with sentence vectors computed by averaging word vectors.

## Workflow and Outputs

- Compute embeddings in Python, or in batch through the text2vec command-line tool, which reads a one-sentence-per-line file and writes a CSV of embeddings. Multi-GPU and multi-CPU inference is supported.
- Score sentence similarity with cosine similarity in the range [-1, 1].
- Run semantic search that returns the top-k most similar sentences from a corpus for a query.
- Models can also be loaded through HuggingFace Transformers or sentence-transformers.

## Training and Models

The repository includes training scripts for supervised and unsupervised CoSENT and SBERT models, BERT-Match, and BGE fine-tuning with (query, positive, negative) triples. Training data can be your own sentence pairs labeled 0/1 or on a 0-5 similarity scale. Released models cover Chinese, multilingual and BGE-based variants, with evaluation tables on ATEC, BQ, LCQMC, PAWSX, STS-B and SOHU sets. Embedding dimensionality reduction and distillation are supported by reusing sentence-transformers methods.

## Deployment

The README describes serving models as a gRPC/HTTP service through Jina or as a native HTTP service with FastAPI. The project also lists released Chinese semantic matching datasets on HuggingFace.

## Features
- Sentence and word embeddings via Word2Vec, SBERT, CoSENT and BGE
- Cosine-based sentence similarity scoring
- Semantic text search over a corpus
- Command-line batch vectorization to CSV
- Multi-GPU and multi-CPU inference
- Supervised and unsupervised model training and fine-tuning
- BGE triplet dataset building with hard negative mining
- Model dimensionality reduction and distillation
- Deployment via Jina gRPC or FastAPI
- Released Chinese and multilingual pretrained models and datasets

## Integrations
PyTorch, HuggingFace Transformers, sentence-transformers, Gensim, Jina, FastAPI, Hugging Face Hub, Gradio

## Platforms
ANDROID, WEB, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
1.3.8

## Links
- Website: https://pypi.org/project/text2vec/
- Documentation: https://github.com/shibing624/text2vec/wiki
- Repository: https://github.com/shibing624/text2vec
- EveryDev.ai: https://www.everydev.ai/tools/text2vec