Kedro

Kedro is an open-source Python framework hosted by the LF AI & Data Foundation. It applies software engineering practices to data engineering and data science code, giving teams a standard project structure, a data connector layer and pipeline tooling. It is installed with pip or conda.

What It Is

Kedro is a toolbox for production-ready data and machine-learning pipelines. A Kedro project provides scaffolding so that exploratory code can move toward reproducible, modular and maintainable code. The README says it was built to address the shortcomings of Jupyter notebooks, one-off scripts and glue code, and to help teams with varied software engineering experience collaborate.

How the Workflow Is Structured

Pipelines are built from pure Python functions, and Kedro resolves dependencies between them from the datasets they consume and produce, so the running order does not need to be labelled by hand. A project template standardises how configuration, source code, tests, documentation and notebooks are organised, and Starters let teams create their own templates. The Data Catalog defines how data is saved and loaded across file formats and file systems, including S3, GCP, Azure, sFTP, DBFS and local storage, and includes versioning for file-based systems.

Visualisation, Tooling and Deployment

Kedro-Viz visualises pipelines, showing data lineage and execution details such as execution time, node status and dataset statistics. A Visual Studio Code extension adds code navigation and autocompletion for Kedro projects. The framework supports test-driven development with pytest, documentation with Sphinx, linting with ruff and standard Python logging. Deployment options include single-machine or distributed runs, with documented support for targets such as Argo, Prefect, Kubeflow, AWS Batch, AWS SageMaker, Databricks and Dask.

Setup Path

The documentation covers installation, key concepts and a spaceflights tutorial for building a first project. Guides also cover working with Jupyter notebooks. The community uses Slack, and the project maintains an awesome-kedro list of extensions, articles and showcase projects.