Infinity

Infinity is an MIT-licensed inference server created by Michael Feil for serving embedding and reranking models behind a REST API. It can deploy models from HuggingFace and is available through pip, Docker images, a Python API and a generated Python client. The latest documented release is 0.0.77 (published 2025-08-22).

What It Is

Infinity is a serving engine for text embeddings, reranking models, CLIP, CLAP and ColPali. Users launch it with the infinity_emb v2 CLI, pointing at one or more HuggingFace model IDs or local model paths. The server exposes an API aligned to OpenAI's embeddings specification, built on FastAPI, with a Swagger UI available at /docs after startup.

Supported Tasks and Models

The README documents several task types, each with tested example models:

• Text embeddings, including decoder-based models
• Reranking with bert-style classification models
• Image-text embeddings via CLIP and SigLIP models
• Audio-text embeddings via CLAP models
• ColBert and ColPali late-interaction embeddings
• Text classification

Plain vision models are documented as not supported, and ColPali models must be merged rather than LoRA adapters.

Inference Backends and Hardware

The server is built on PyTorch, optimum (ONNX/TensorRT) and CTranslate2, with FlashAttention. It targets NVIDIA CUDA, AMD ROCm, CPU, AWS INF2 and Apple MPS accelerators, and uses dynamic batching with tokenization in dedicated worker threads. Specialized Docker images exist for CPU, ROCm and ONNX/TensorRT.

Deployment and Configuration

Multiple models can be launched in one server by repeating CLI options. All CLI arguments can also be set through INFINITY_-prefixed environment variables, and an API key can be set via --api-key. Telemetry can be disabled with an environment variable. The README lists integrations such as LangChain, Runpod, KubeAI, RAGFlow and TrueFoundry Cognita.