llm-d

llm-d is an open-source distributed inference serving stack for Kubernetes, hosted as a CNCF Sandbox project and founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. It runs model servers such as vLLM and SGLang across a cluster and adds routing, cache management, and scaling on top. The latest release listed on GitHub is v0.10.0.

What It Is

llm-d sits above model servers and turns single-node engines into production-grade distributed inference on infrastructure teams already run. It is described as engine- and hardware-agnostic, targeting GPUs, Google TPUs, Intel XPUs, and CPUs, and is aimed at agentic pipelines, LLMs, multimodal models, and high-throughput serving.

Well-Lit Paths

The project ships Well-Lit Paths: tested, benchmarked deployment recipes for common production patterns. Each path includes Helm charts and Kustomize manifests, key tuning knobs, sample workloads and benchmarks against baselines, and monitoring and observability configuration. Paths cover an optimized baseline, predicted-latency routing, precise prefix-cache-aware routing, tiered prefix cache, prefill/decode disaggregation, wide expert parallelism for MoE models, flow control and fairness, and workload autoscaling.

Capabilities by Theme

• Intelligent routing with prefix-cache and load-aware balancing, plus predicted-latency scheduling.
• KV-cache management with tiered offloading to CPU or disk and global indexing of cache state.
• Serving large models through prefill/decode disaggregation and wide expert parallelism.
• Operational controls such as multi-tenant flow control and SLO-aware autoscaling.
• Batch processing through OpenAI-compatible Batch APIs and asynchronous processing.

Performance Claims

The README attributes results to partner benchmarks and blogs, for example 3x higher output throughput and 2x faster TTFT with prefix-cache-aware routing versus round-robin in a Red Hat/Tesla test, and up to 70% higher tokens/sec with disaggregation in an AWS test. Reproducible benchmarks are published on the Prism site.