Kubeflow Trainer

Kubeflow Trainer is an open-source, Kubernetes-native platform from the Kubeflow community for distributed AI model training and LLM fine-tuning. It exposes a single TrainJob custom resource and a Python SDK that work across PyTorch, JAX, DeepSpeed, MLX, HuggingFace, Megatron, and XGBoost. The latest release listed in the repository is v2.3.0, and the README describes the project as alpha status with APIs that may change.

What It Is

Kubeflow Trainer lets AI practitioners define training jobs as TrainJobs that run on Kubernetes clusters, while platform administrators supply reusable training runtimes. A job can be written once with the Python SDK, for example a CustomTrainer with a training function and a node count, and run locally with Docker or Podman or on a multi-node GPU cluster. It is a Kubeflow subproject and a CNCF project.

Distributed Training and Fine-Tuning

The documentation describes automatic setup of DDP, FSDP, parameter servers, and gang-scheduling across nodes. It also brings MPI to Kubernetes for multi-node, multi-GPU jobs on HPC-style clusters. For LLM fine-tuning, it supports LoRA, QLoRA, and full fine-tuning through a TorchTune built-in trainer, using HuggingFace model and dataset URIs. A distributed data cache built on Apache Arrow and Apache DataFusion streams data to GPU nodes with zero-copy transfer.

Operator Controls and Scheduling

Operator guides cover installation, runtimes, ML policy, job templates, runtime patches, Dynamic Resource Allocation, and an extension framework for customizing scheduling and resource management. Documented scheduling integrations include Kueue, Slurm Bridge, KAI Scheduler, coscheduling, and Volcano. Under the hood it reuses JobSet and LeaderWorkerSet.

Recent Releases

The README lists v2.3.0 (2026-08), which adds a runtime snapshot mechanism for decoupled runtime lifecycle, enhanced MPI support, and Megatron-LM examples. Version 2.2 added JAX and XGBoost runtimes, metrics propagation to TrainJob status, and Flux Framework integration. Migration guidance and archived documentation exist for the legacy Training Operator v1.