# LoRAX

> Multi-LoRA inference server for serving thousands of fine-tuned LLMs on a single GPU.

LoRAX (LoRA eXchange) is an open-source inference server from Predibase that serves many fine-tuned LoRA adapters on top of a shared base model on a single GPU. Requests name an adapter, which is loaded just-in-time. The GitHub repository is marked as archived, and the latest listed release is lorax-0.4.0 (January 2025).

## What It Is

LoRAX is a framework for serving fine-tuned large language models. A pretrained base model is shared across all adapters, and task-specific adapter weights are loaded per request. Adapters can come from HuggingFace, Predibase, or a local filesystem. The project says this reduces serving cost without compromising throughput or latency. It is built on HuggingFace's text-generation-inference, forked from v0.9.4.

## How Serving Works

The server is typically launched from a prebuilt Docker image with a base model ID. Users then prompt it through a REST `/generate` endpoint, the `lorax-client` Python client, or an OpenAI-compatible chat API, where the adapter is passed as the `model` parameter. Adapters can also be merged per request to create ensembles.

## Performance and Production Features

- Heterogeneous continuous batching packs requests for different adapters into the same batch.
- Adapter exchange scheduling prefetches and offloads adapters between GPU and CPU memory.
- Optimizations include tensor parallelism, flash-attention, paged attention, SGMV kernels, quantization, and token streaming.
- Operational support includes Helm charts for Kubernetes, Prometheus metrics, OpenTelemetry tracing, per-request tenant isolation for private adapters, and structured JSON output.

## Models and Requirements

Supported base models include Llama (including CodeLlama), Mistral (including Zephyr), and Qwen. Base models can be loaded in fp16 or quantized with bitsandbytes, GPT-Q, or AWQ. Supported adapters are LoRA adapters trained with PEFT and Ludwig. The documented minimum requirements are an Nvidia Ampere-generation or newer GPU, CUDA 11.8-compatible drivers, Linux, and Docker for the Docker guide.

## Features
- Dynamic adapter loading from HuggingFace, Predibase, or local filesystem
- Heterogeneous continuous batching across adapters
- Adapter exchange scheduling between GPU and CPU memory
- Tensor parallelism and pre-compiled CUDA kernels (flash-attention, paged attention, SGMV)
- Quantization with bitsandbytes, GPT-Q, and AWQ
- Token streaming
- OpenAI-compatible API with multi-turn chat
- Per-request adapter merging
- Structured output (JSON mode)
- Per-request tenant isolation for private adapters
- Prometheus metrics and OpenTelemetry tracing
- Prebuilt Docker images and Helm charts
- Speculative decoding and CUDA graph compilation guides

## Integrations
HuggingFace, Predibase, PEFT, Ludwig, Docker, Kubernetes (Helm), SkyPilot, Prometheus, OpenTelemetry, OpenAI API

## Platforms
LINUX, API, CLI, DEVELOPER_SDK

## Pricing
Open Source

## Version
lorax-0.4.0

## Links
- Website: https://loraexchange.ai
- Documentation: https://loraexchange.ai
- Repository: https://github.com/predibase/lorax
- EveryDev.ai: https://www.everydev.ai/tools/lorax