Data Versioning Explained the Open Source Way
summary
Surveys four concrete approaches to data versioning and when to pick each: ML pipeline tools (DVC and similar) version training sets, hyperparameters and models for reproducibility; collaboration platforms (Dolt, Kaggle, DoHub) add shared datasets, git-like branching and merge workflows; mutable table formats (Delta Lake, Iceberg, Hudi over Parquet) implement metadata-backed mutability and time travel for per-table history; and an object-store layer (lakefs) exposes a branching model and copy-on-write pointers so teams run isolated experiments, CI/CD checks and atomic merges.
AI-written summary · about
chapters (6)
- 0:00Introduction and background
- 3:09MLOps data versioning tools
- 6:17Collaboration-focused versioning tools
- 11:15Object storage and mutable formats
- 16:11Git-like versioning for data lakes
- 20:55Q&A and closing remarks
related talks
36mData Versioning - does it mean what we think it means?
24mModel Versioning: Why, When, and How
31mGetting ML Right in a Complex Data World
36mTable Formats Change Everything (By Not Changing Anything)
16mMake your data science workflows reproducible
47mKeynote - The Evolving Role of Data in Software Development
27mRevision Control for Structured Data
26mCI/CD For Data – Building Data Development Environments With Open Source Stacks