Data versioning in machine learning projects
summary
Open-source DVC claims to make experiments reproducible by pairing code with the exact data version and pipeline commands rather than just restating the topic. It works on top of Git and stores meta-files not data, pushing large files to remotes (s3 GCS HDFS). It tracks inputs, outputs and metrics, builds dependency graphs for pipelines, and optimizes checkouts with hard links so switching branches restores the correct gigabyte-scale datasets quickly.
AI-written summary · about
chapters (21)
- 0:00Data versioning in ML
- 1:56Data vs code versioning
- 4:34ML workflows vs software
- 7:56Introducing Data Ocean Control
- 9:01Data separate from code
- 10:10Git LFS limitations
- 11:59Data Ocean file handling
- 13:48Syncing data remotely
- 14:31Optimizing data checkout
- 16:16ML pipelines defined
- 17:42Reproducibility in pipelines
- 18:28ML workflow vs Git flow
- 20:43Tracking failed experiments
- 23:47Adopting ML platform
- 25:18Open-source ML needs
- 26:45Q&A: Tool unification
- 27:50Q&A: Workflow focus
- 28:53Q&A: Schema changes
- 29:59Q&A: Merge conflicts
- 31:58Q&A: Data deduplication
- 33:49Q&A: Integration with data engineering
related talks
36mData Versioning - does it mean what we think it means?
37mOpen Source Tools for ML Experiments Management
59mDevOps for Machine Learning Projects
01h 32mHow to easily set up and version control your Machine Learning Pipelines
01h 25mBest practices and solution design using data versioning for machine learning
32mReproducibility and versioning of ML systems
27mLeveraging Git for ML Experiment Management
02h 10mBuild End-To-End MLOps Platform with Open Source DVC Ecosystem