# Infinity

> Open-source high-throughput, low-latency REST API server for serving text embeddings, reranking, CLIP, CLAP and ColPali models.

Infinity is an MIT-licensed inference server created by Michael Feil for serving embedding and reranking models behind a REST API. It can deploy models from HuggingFace and is available through pip, Docker images, a Python API and a generated Python client. The latest documented release is 0.0.77 (published 2025-08-22).

## What It Is

Infinity is a serving engine for text embeddings, reranking models, CLIP, CLAP and ColPali. Users launch it with the `infinity_emb v2` CLI, pointing at one or more HuggingFace model IDs or local model paths. The server exposes an API aligned to OpenAI's embeddings specification, built on FastAPI, with a Swagger UI available at `/docs` after startup.

## Supported Tasks and Models

The README documents several task types, each with tested example models:

- Text embeddings, including decoder-based models
- Reranking with bert-style classification models
- Image-text embeddings via CLIP and SigLIP models
- Audio-text embeddings via CLAP models
- ColBert and ColPali late-interaction embeddings
- Text classification

Plain vision models are documented as not supported, and ColPali models must be merged rather than LoRA adapters.

## Inference Backends and Hardware

The server is built on PyTorch, optimum (ONNX/TensorRT) and CTranslate2, with FlashAttention. It targets NVIDIA CUDA, AMD ROCm, CPU, AWS INF2 and Apple MPS accelerators, and uses dynamic batching with tokenization in dedicated worker threads. Specialized Docker images exist for CPU, ROCm and ONNX/TensorRT.

## Deployment and Configuration

Multiple models can be launched in one server by repeating CLI options. All CLI arguments can also be set through `INFINITY_`-prefixed environment variables, and an API key can be set via `--api-key`. Telemetry can be disabled with an environment variable. The README lists integrations such as LangChain, Runpod, KubeAI, RAGFlow and TrueFoundry Cognita.

## Features
- REST API for text embeddings, reranking, CLIP, CLAP and ColPali
- Deploy any compatible HuggingFace model
- Multiple models served from one instance
- Dynamic batching with dedicated tokenization worker threads
- PyTorch, ONNX/TensorRT and CTranslate2 backends
- OpenAI-compatible embeddings API specification
- Late-interaction ColBert and ColPali embeddings
- Text classification
- API key support
- Configuration via CLI or environment variables
- Python async API and generated Python client
- Docker images for CUDA, CPU, ROCm and ONNX/TensorRT
- Swagger UI
- Optional telemetry opt-out

## Integrations
HuggingFace, LangChain, Runpod, TrueFoundry Cognita, KubeAI, RAGFlow, SAP AI Core, Modal, PyTorch, ONNX, TensorRT, CTranslate2, Qdrant, Vespa

## Platforms
API, CLI, DEVELOPER_SDK, LINUX

## Pricing
Open Source

## Version
0.0.77

## Links
- Website: https://michaelfeil.github.io/infinity/
- Documentation: https://michaelfeil.github.io/infinity/
- Repository: https://github.com/michaelfeil/infinity
- EveryDev.ai: https://www.everydev.ai/tools/infinity