lakeFS

lakeFS is a scalable data version control system built by Treeverse that applies proven software engineering practices to data lake management. It enables teams to branch, merge, commit, and roll back data just like code, providing isolated environments for testing, reproducible ML experiments, and atomic data promotion. Trusted by organizations like Netflix, Volvo, Lockheed Martin, and Amazon, lakeFS integrates with virtually every major data and AI stack without moving data out of your storage. It is available as an open-source project and as a managed Enterprise offering with advanced security and governance features.

• Data Branching & Merging: Create zero-copy branches of your data lake for isolated testing and experimentation, then atomically merge changes back to production.
• Format-Agnostic Version Control: Works with any data format—Parquet, CSV, Avro, JSON, Delta Lake, Iceberg, Hudi, and unstructured data like images and video.
• Data CI/CD with Hooks: Enforce data quality and compliance standards automatically using lakeFS hooks before changes reach production.
• Instant Rollback: Recover from data incidents immediately by reverting to any previous commit without duplicating data.
• Audit Trail & Lineage: Gain full visibility into data history with built-in audit logs to satisfy model governance and compliance requirements.
• Role-Based Access Control (RBAC): Enterprise plan includes RBAC, SSO, SCIM, and IAM Roles for fine-grained, secure access management across teams.
• lakeFS Mount: Virtually mount remote lakeFS repositories as a local filesystem for high-performance deep learning workloads.
• Transactional Mirroring: Replicate repositories to remote regions for disaster recovery and data locality without data inconsistency.
• Broad Integrations: Connects natively with Spark, Databricks, Airflow, Kafka, Flink, Airbyte, dbt, MLflow, Kubeflow, AWS SageMaker, and many more tools.
• Cloud & Storage Agnostic: Supports AWS S3, Azure Blob, Google Cloud Storage, MinIO, Ceph, Dell EMC, and on-premises storage via the S3 interface.

To get started, run lakeFS locally using the quickstart guide at docs.lakefs.io, or sign up for lakeFS Cloud. Connect your existing object storage, create a repository, and begin branching your data just like a Git workflow.