LLM2Vec

LLM2Vec is an open-source project from McGill NLP that turns decoder-only large language models into text encoders. It ships as a Python package (llm2vec) with training scripts, evaluation tooling, and a collection of pretrained model checkpoints on Hugging Face. The latest listed release is version 0.2.3.

What It Is

LLM2Vec is a three-step recipe: enable bidirectional attention, train with masked next token prediction (MNTP), and apply unsupervised contrastive learning (SimCSE). The resulting models can be fine-tuned further with supervised contrastive training. The LLM2Vec class wraps Hugging Face models to handle bidirectionality, sequence encoding, and pooling.

How the Workflow Runs

Users load a base model plus an optional PEFT/LoRA adapter through from_pretrained, then call encode on text or on [instruction, text] pairs to get embeddings. Mean pooling is the default and can be changed, and the maximum sequence length defaults to 512. Embeddings can be compared with cosine similarity for retrieval, classification, clustering, and sentence similarity tasks, with examples in the repository.

Training and Evaluation

The repository includes scripts for MNTP training, SimCSE training on Wikipedia sentences, supervised contrastive training on public E5 data, and word-level tasks such as POS, NER, and chunking. Configuration files cover Meta-Llama-3-8B, Mistral-7B, Llama-2-7B, and Sheared-Llama-1.3B. MTEB evaluation is supported through an evaluation script and per-task instructions.

Recent Updates

The README notes support for newer transformers versions covering Llama 3.1 and 3.2, plus Gemma and Qwen-2 models. It also announces LLM2Vec-Gen, a separate recipe for generative embeddings that encode a potential answer to a query.