# AnglE

> Python library for training and running BERT- and LLM-based sentence embeddings using angle-optimized loss.

AnglE is an open-source Python library from the paper "AnglE-optimized Text Embeddings" (ACL 2024). It lets users train sentence embedding models and run inference with a variety of transformer-based embedding models. It is distributed as the `angle-emb` package under the MIT License, and the latest listed release is v0.6.0.

## What It Is

AnglE is a sentence-embedding training and inference framework. Its core idea, per the paper, is angle optimization in a complex space, which addresses the vanishing-gradient problem caused by the saturation zones of the cosine function in embedding objectives. The library supports BERT-based models (BERT, RoBERTa, ModernBERT), LLM-based models (LLaMA, Mistral, Qwen) and bidirectional LLMs via BiLLM.

## Training and Loss Options

Users prepare a Hugging Face dataset in one of three formats: paired texts with similarity scores, query-positive pairs, or query-positive-negative triples. Training runs through the `angle-trainer` CLI or a Python `fit` API, on a single GPU or multiple GPUs (including FSDP with accelerate). Supported losses are AnglE, contrastive, CoSENT and Espresso (2DMSE). Options include LoRA for LLMs, prompt templates, column renaming for older datasets, and teacher or self-distillation to reduce catastrophic forgetting. Trained models can be converted to sentence-transformers format.

## Inference and Pretrained Models

`AnglE.from_pretrained` loads official models, such as WhereIsAI/UAE-Large-V1 for general English, UAE-Code-Large-V1 for code similarity and the pubmed-angle models for medical similarity. It can also load third-party transformer embedding models. Prompts can be applied for retrieval tasks. Espresso/Matryoshka models support layer truncation and reduced embedding sizes, and dynamic batching is available through the `batched` library.

## Research Context

The README reports that the UAE-Large-V1 model and mixedbread's mxbai-embed-large-v1, both trained with AnglE, reached top scores on the MTEB Leaderboard at the time of those entries. The paper reports that AnglE outperforms state-of-the-art STS models that ignore the cosine saturation zone.

## Features
- AnglE, contrastive, CoSENT and Espresso (2DMSE) loss functions
- BERT-based, LLM-based and BiLLM backbones
- Single-GPU and multi-GPU (FSDP) training
- CLI trainer and Python fit API
- Three dataset formats for similarity, retrieval and contrastive training
- Official pretrained embedding models on Hugging Face
- Espresso/Matryoshka layer and dimension truncation
- Prompt support for queries and documents
- Inference with third-party transformer embedding models
- Dynamic batching for large-scale inference
- Conversion to sentence-transformers format
- Knowledge and self-distillation to prevent catastrophic forgetting

## Integrations
Hugging Face, sentence-transformers, PyTorch, Accelerate, LoRA, BiLLM, MTEB, Weights & Biases, Google Colab

## Platforms
LINUX, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v0.6.0

## Links
- Website: https://arxiv.org/abs/2309.12871
- Documentation: https://angle.readthedocs.io/en/latest/index.html
- Repository: https://github.com/SeanLee97/AnglE
- EveryDev.ai: https://www.everydev.ai/tools/angle