CI/CD For Data – Building Data Development Environments With Open Source Stacks
summary
Claims a git-like layer (lakeFS) over object stores that does zero-copy branching of S3/ADLS/GCS objects, hashes objects to detect changes, and uses copy-on-write so only about 20% of bytes are rewritten after compute. Provides hooks/webhooks and YAML actions to run Spark jobs, Great Expectations, or custom checks before merges, blocks bad formats (CSV→Parquet rule in demo), and lets teams revert to exact commits for fast rollback and audit.
AI-written summary · about
chapters (8)
- 0:00Start
- 2:52Maintaining Legacy Code
- 5:35Data Life Cycle
- 8:35Deployment
- 10:15Version Control
- 11:18Git for Data
- 13:29Copy on Write
- 19:24Demo
related talks
08mA Git-like Repository for Your Data Lake
31mGetting ML Right in a Complex Data World
33mBuilding Reproducible ML Processes with an Open Source Stack
33mThe advent of the open data lake
31mHow to Implement DataOps using
22mData Exploration Made Easy with Open Source
30mDevOps For Data Teams
39mDelta Lake: Diving into Data Lakes Without the Downsides