CI/CD for Data - Building Dev/Test Data Environments with Open Source Stacks
summary
Demonstrates a CI/CD approach that versions an entire data lake with lakeFS (git-like API over S3/MinIO/Azure/GCS), creates isolated branches in milliseconds via pointer copy-on-write for petabyte lakes, and runs pre-merge hooks (YAML + Python webhooks) to gate merges with Great Expectations / Soda or custom validators. Shows simple one-line time travel reverts for rollbacks, branch-protected promotion workflows, and configurable garbage-collection/retention (example: 30-day soft delete) to reduce storage and testing risk.
AI-written summary · about
chapters (9)
- 0:00CI/CD for data pipelines
- 2:28Why pipelines break
- 7:05Software CI/CD for data
- 8:29LakeFS for versioning
- 11:41Test environments with branches
- 13:25LakeFS hooks for quality
- 14:28Deploying data with CI/CD
- 16:57Rolling back data issues
- 18:24Summary and resources
related talks
08mA Git-like Repository for Your Data Lake
38mML Data Version Control and Reproducibility at Scale
31mFrom Warehouse to Lakehouse: Data at Rest
25mBuild open format data lakehouse
37mBuild a Modern Data Platform on AWS
01h 03mReinvente su estrategia de datos con Lagos de Datos Transaccionales
29mData Acquisition: Overcoming Challenges
01h 28mBuilding Reliable, Open Lakehouses with Delta Lake