Kubeflow

Kubeflow is an open source, Kubernetes-native AI platform maintained by The Kubeflow Authors as a Cloud Native Computing Foundation project. The homepage announces that it is now a CNCF graduated project. It is composed of modular subprojects covering interactive notebooks, distributed training, hyperparameter tuning, ML pipelines, model metadata management and a central dashboard.

What It Is

Kubeflow is a stack for data and AI/ML workloads on Kubernetes. It is aimed at AI practitioners, platform administrators and decision-makers who want to run workloads at scale without becoming Kubernetes experts. Its stated mission is to bridge the Data, AI and Cloud Native ecosystems and provide paths across the AI lifecycle for models, agents and AI applications.

Modular Subprojects

• Kubeflow Notebooks runs interactive development environments for AI, ML and data workloads on Kubernetes.
• Kubeflow Trainer is a distributed platform for LLM fine-tuning and model training across frameworks including PyTorch, MLX, HuggingFace, DeepSpeed, Megatron, JAX and XGBoost.
• Kubeflow Katib provides AutoML with hyperparameter tuning, early stopping and neural architecture search.
• Kubeflow Pipelines builds and deploys portable, scalable ML workflows.
• Kubeflow Hub (formerly Model Registry) indexes and manages models, versions and artifact metadata.
• Kubeflow Spark Operator makes running Spark applications on Kubernetes idiomatic.
• Kubeflow Dashboard connects the authenticated web interfaces of Kubeflow and ecosystem components.

Design Principles

The project lists four principles: simple, portable (same code on a laptop, on-premises or any cloud), scalable, and composable, so teams can mix and match tools across the AI lifecycle.