Data Versioning - does it mean what we think it means?
summary
Compares concrete tools and use cases: DVC and MLflow for versioning data+code+configs in ML pipelines; Iceberg, Hudi, Delta Lake for adding mutability to object-storage lakes (adds metadata-driven deltas, usually short time travel windows of weeks/months); collaboration stacks like DoltHub and Kaggle for shared datasets; and a built Git-like layer (lakeFS) that provides branching and atomic commits over S3 without copying data, proven at petabyte scale with hundreds of millions of objects but often assuming a single writer and incurring metadata management overhead.
AI-written summary · about
chapters (9)
- 0:00Introduction to talk
- 0:42Defining data versioning
- 3:30Clustering tools by use case
- 7:19Managing ML pipelines
- 13:00Collaboration over data
- 18:47Mutability in data lakes
- 25:45Data lake management challenges
- 27:07Git-like capabilities for data
- 32:01Q&A session
related talks
24mData Versioning Explained the Open Source Way
35mData versioning in machine learning projects
31mGetting ML Right in a Complex Data World
24mModel Versioning: Why, When, and How
38mML Data Version Control and Reproducibility at Scale
54mThe Future of Data Science/ML at Scale: A Look at MLflow, Delta Lake + Lakehouse
28mVersioning Multimodal Data: Metadata & Beyond
16mThe Evolution of Delta Lake