Kueue

Kueue is an open-source set of APIs and a controller from the Kubernetes community (SIG Scheduling, hosted under kubernetes-sigs) for job-level queueing. It decides when a job should be admitted to start and when it should stop, and it works with common Kubernetes workload types. The project lists v0.20.1 as its latest release.

What It Is

Kueue is a job-level manager that acts as an admission controller before the Kubernetes scheduler, managing queues and quotas without replacing core Kubernetes components. Administrators define quotas and queues with native Kubernetes APIs, and Kueue admits workloads (so pods can be created) or evicts them (so active pods are deleted) according to those rules. It targets batch, HPC, and AI/ML workloads.

Quota and Resource Sharing

Quotas are defined at the cluster level and shared across teams and namespaces. Queues can be grouped into cohorts so workloads borrow idle capacity from peers. Queueing strategies include StrictFIFO and BestEffortFIFO, and priority-based preemption and Fair Sharing apply between tenants. Resource flavor fungibility lets workloads route to different hardware pools, such as Spot versus On-Demand instances or different GPU models.

Workload Integrations and Placement

Built-in integrations cover Kubernetes Job, JobSet, Kubeflow training jobs, RayJob, RayCluster, LeaderWorkerSet, plain Pods and Pod Groups, and serving workloads such as Deployments and StatefulSets. Topology-Aware Scheduling groups tightly coupled pods based on data-center topology to improve pod-to-pod communication. MultiKueue dispatches jobs across multiple worker clusters from a central control plane. Dynamic Resource Allocation support manages quota for devices such as GPUs.

Operations and Setup

AdmissionChecks let internal or external components influence admission, including integration with cluster-autoscaler ProvisioningRequests. Other features include all-or-nothing scheduling with ready pods, partial admission with reduced parallelism, and dynamic reclaim of quota. Prometheus metrics and an on-demand visibility endpoint for pending workloads support monitoring. Kueue is installed by applying a release manifest with kubectl, runs in the kueue-system namespace, and is officially supported and tested on Kubernetes 1.34 or newer.