LoRAX

LoRAX (LoRA eXchange) is an open-source inference server from Predibase that serves many fine-tuned LoRA adapters on top of a shared base model on a single GPU. Requests name an adapter, which is loaded just-in-time. The GitHub repository is marked as archived, and the latest listed release is lorax-0.4.0 (January 2025).

What It Is

LoRAX is a framework for serving fine-tuned large language models. A pretrained base model is shared across all adapters, and task-specific adapter weights are loaded per request. Adapters can come from HuggingFace, Predibase, or a local filesystem. The project says this reduces serving cost without compromising throughput or latency. It is built on HuggingFace's text-generation-inference, forked from v0.9.4.

How Serving Works

The server is typically launched from a prebuilt Docker image with a base model ID. Users then prompt it through a REST /generate endpoint, the lorax-client Python client, or an OpenAI-compatible chat API, where the adapter is passed as the model parameter. Adapters can also be merged per request to create ensembles.

Performance and Production Features

• Heterogeneous continuous batching packs requests for different adapters into the same batch.
• Adapter exchange scheduling prefetches and offloads adapters between GPU and CPU memory.
• Optimizations include tensor parallelism, flash-attention, paged attention, SGMV kernels, quantization, and token streaming.
• Operational support includes Helm charts for Kubernetes, Prometheus metrics, OpenTelemetry tracing, per-request tenant isolation for private adapters, and structured JSON output.

Models and Requirements

Supported base models include Llama (including CodeLlama), Mistral (including Zephyr), and Qwen. Base models can be loaded in fp16 or quantized with bitsandbytes, GPT-Q, or AWQ. Supported adapters are LoRA adapters trained with PEFT and Ludwig. The documented minimum requirements are an Nvidia Ampere-generation or newer GPU, CUDA 11.8-compatible drivers, Linux, and Docker for the Docker guide.