AIBrix

AIBrix is an open-source Kubernetes control plane for production LLM inference, hosted under the vllm-project GitHub organization. Users bring their own engine (vLLM, SGLang, or TensorRT-LLM), and AIBrix routes requests using live engine state, scales on inference metrics, and manages models, LoRA adapters, and KV cache. The latest release listed is v0.8.0.

What It Is

AIBrix sits between clients and inference engine pods on Kubernetes. Requests hit an OpenAI-compatible gateway and go straight to an engine pod, while Kubernetes controllers decide what should be running: model lifecycle, warmup, multi-node topology, autoscaling, and observability. It is aimed at teams running many models or replicas, mixing GPU types, or needing to hold cost and latency targets.

Routing and Serving Capabilities

The gateway picks a pod based on live engine state, including prefix-cache, load, and prefill/decode placement, with session affinity for agentic workloads. Multiple engines can be served behind one gateway. Many low-rank LoRA adapters can be packed onto shared base-model pods. Models are declared with ModelClaim, and ModelWarmup preloads images and weights so scale-up is faster. A runtime sidecar downloads weights, loads and unloads adapters, and normalizes engine metrics.

Scaling and Distributed Inference

Autoscaling acts on engine metrics such as queue depth, KV cache, and tokens. StormService runs multi-node and multi-role deployments, including prefill/decode. A distributed KV cache offloads cache and reuses it across engines. Heterogeneous GPU support, to mix GPU types while targeting cost and SLO, is marked experimental.

Setup Path

Installation applies release manifests with kubectl for dependencies (Envoy Gateway, KubeRay), CRDs, and core controllers into the current cluster context; KubeRay is only required for Ray-based multi-node inference. Helm steps and a quickstart with a base-model manifest and OpenAI-compatible curl commands are in the documentation. It can also be installed from a repository checkout using kustomize, with images tagged nightly.

Release History

The README lists releases from v0.1.0 through v0.8.0, with blog posts for most, and conference talks including KubeCon keynotes.