ML Data Version Control and Reproducibility at Scale
summary
Demonstrates a Git-like system (lakeFS) that version-controls S3/Minio data with zero-copy pointers so petabytes don’t get duplicated, supports branches that resolve to commit IDs in milliseconds, and replays exact data+code experiments via MLflow links. Demo ingests the Airbus ship dataset (~200,000 images), runs a local proof with 100 images and a full pipeline using Delta Lake + PyTorch Lightning, records metrics and artifacts, and shows garbage collection reclaims unneeded object versions, proving the approach scales and works.
AI-written summary · about
chapters (7)
- 0:00Data reproducibility at scale
- 10:18Demo setup prerequisites
- 12:08Computer vision project walkthrough
- 25:44Local data processing with LakeFS
- 30:05Medallion architecture in practice
- 35:23Model training Git integration
- 37:13Q&A
related talks
31mGetting ML Right in a Complex Data World
32mReproducibility and versioning of ML systems
54mThe Future of Data Science/ML at Scale: A Look at MLflow, Delta Lake + Lakehouse
08mA Git-like Repository for Your Data Lake
36mData Versioning - does it mean what we think it means?
01h 25mBest practices and solution design using data versioning for machine learning
33mCompliant Data Management and Machine Learning on Kubernetes
20mCI/CD for Data - Building Dev/Test Data Environments with Open Source Stacks