llm-d
llm-d is an open-source, Kubernetes-native stack for distributed large-language-model inference. It turns single-node inference engines such as vLLM and SGLang into production-grade serving across clusters, models, workloads and heterogeneous accelerators, with the stated vision of any model, any accelerator, any cloud.
At a Glance
- Enterprise platform-engineering and AI-infrastructure teams
- Cloud providers and managed Kubernetes operators
- Model providers and AI accelerator vendors
- Organizations self-hosting proprietary or open-weight models
- +1 more
AI Tools by llm-d
(1)
llm-d
Kubernetes LLM Inference Serving Stack
Discussions
No discussions yet
Be the first to start a discussion about llm-d
Latest News
Products & Services
Open-source Kubernetes-native control and serving stack that runs vLLM, SGLang and other engines across clusters for production distributed inference.
Intelligent entry point and routing layer for inference requests, including prefix-cache-aware, predicted-latency, token-aware and flow-control routing.
Distributed KV-cache indexing, routing and offloading capabilities, including multi-tier storage and integrations built around LMCache and vLLM connectors.
Tested and benchmarked deployment recipes with Helm/Kustomize artifacts, tuning knobs, sample workloads, benchmarks and monitoring configuration for common production patterns.
Market Position
llm-d positions itself as a vendor-neutral, Kubernetes-native distributed-inference control plane rather than a single model server or single-cloud service. It is differentiated by combining vLLM/SGLang with AI-aware routing, prefix-cache and KV-cache management, prefill/decode disaggregation, tested Well-Lit Paths and broad accelerator portability; relevant alternatives and adjacent competitors include KServe, NVIDIA Dynamo, Ray Serve, Hugging Face TGI, SGLang, BentoML and managed cloud inference platforms.
Leadership
Founders
Red Hat
Open-source enterprise technology company that launched the llm-d community at Red Hat Summit and champions its growth.
Google Cloud
Cloud and AI infrastructure provider; a founding contributor bringing Kubernetes, GKE, TPU and AI Hypercomputer expertise.
IBM Research
IBM's research organization; a founding contributor focused on hybrid-cloud AI platforms and distributed inference.
CoreWeave
AI cloud provider and founding contributor, bringing production AI-infrastructure and inference-at-scale experience.
NVIDIA
AI accelerator and software company; a founding contributor supporting scalable inference and technologies including NIXL.
Founding Story
llm-d was launched in May 2025 as a collaborative open-source response to the difficulty and cost of running increasingly large generative-AI models in production. The initial vision was a vendor-neutral, Kubernetes-native inference stack that would make distributed serving predictable and efficient across any model, accelerator and cloud, building on vLLM, Kubernetes and intelligent AI-aware routing.
Business Model
Revenue Model
Open-source Apache 2.0 infrastructure project distributed through public source repositories, documentation and community support; no paid subscription or usage-pricing model was identified.
Target Markets
- Enterprise platform-engineering and AI-infrastructure teams
- Cloud providers and managed Kubernetes operators
- Model providers and AI accelerator vendors
- Organizations self-hosting proprietary or open-weight models
- Research and post-training teams running high-throughput or reinforcement-learning workloads
- Production serving of open-weight LLMs and multimodal models
- High-throughput interactive chat, completion and agentic workloads
- Long-context inference that benefits from prefix caching and KV-cache offload
- Prefill/decode-disaggregated serving
- Batch inference, embeddings, synthetic-data creation and offline document processing
- Multi-tenant inference with fairness, admission control and SLO-aware routing