GPUStack

GPUStack is an open-source GPU cluster manager for AI model serving and on-demand GPU instance provisioning. It configures and orchestrates inference engines such as vLLM, SGLang and TensorRT-LLM across on-premise, Kubernetes and cloud GPU environments, and exposes models through OpenAI-compatible and Anthropic-compatible APIs. The latest release listed on GitHub is v2.2.3, and the project is licensed under Apache 2.0.

What It Is

GPUStack is a management layer that sits on top of inference engines and GPU hardware. Teams connect GPU workers to a GPUStack server, deploy models from a catalog or from Hugging Face, ModelScope or local files, and serve them through standard APIs. It also launches SSH-accessible GPU instances for development, fine-tuning and interactive workloads, with Jupyter Notebook access and persistent storage.

How the Workflow Runs

The server can run on a CPU-only machine via a single Docker command, and workers are added from the web UI by running a generated Docker command on each GPU node. Deployment includes automated compatibility checks, and GPUStack maps hardware to a matching inference engine version. Large models can be distributed across nodes and GPUs using tensor and pipeline parallelism. Performance modes (throughput, latency, standard, custom), KV cache extensions such as LMCache and HiCache, and speculative decoding methods like EAGLE3, MTP and N-grams are supported. The vendor reports tuned-deployment gains over unoptimized vLLM baselines in its Performance Lab.

Hardware and Model Coverage

Supported accelerators include NVIDIA, AMD, Ascend NPU, Hygon DCU, Moore Threads, MetaX, Cambricon MLU, Iluvatar and T-Head PPU. Worker nodes run on Linux only; macOS is not supported for workers. Supported model types include LLM, multimodal, embedding, reranker, image, speech and OCR models.

Governance and Enterprise Edition

The platform includes user authentication, API key management, token quotas and rate limits, usage analytics, metering, and Prometheus and Grafana monitoring. The Enterprise Edition adds control plane and model service high availability, multi-tenancy with organization isolation, RBAC with LDAP/OIDC/SAML SSO, audit logs, IP allow and block lists, resource topology view, billing reports and white-label branding.