Getting ML Right in a Complex Data World
summary
A data versioning engine called LakeFS provides git-like data versioning on top of S3-backed object stores (also works with MinIO), letting teams branch, tag and commit datasets, features, configs, models and metrics atomically. Demo uses the wine quality dataset, shows branching is metadata-only (no deep copies), records experiments where one branch yields 64 F1 -> 88 F1, merges the winning branch to main, and reproduces the exact iteration from a tag.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:02MLOps ecosystem and iterative model dev
- 3:28ML experimentation challenges
- 6:00Gaps in MLOps tools
- 7:22Lake FS for ML experimentation
- 9:54Lake FS architecture and data versioning
- 12:45Collaboration and version control in ML
- 14:53Other use cases beyond ML experimentation
- 16:18Live demo setup
- 17:21Create Lake FS repo and branch
- 18:47Upload and commit training data
- 20:12Data exploration with DuckDB
- 21:38Run first ML experiment
- 24:16View commit history and lineage
- 25:20Run second ML experiment
- 26:53Merge experiments to production
- 27:58Reproduce experiments with tags
- 28:41Open source and community
- 29:26Q&A
related talks
38mML Data Version Control and Reproducibility at Scale
36mData Versioning - does it mean what we think it means?
24mData Versioning Explained the Open Source Way
08mA Git-like Repository for Your Data Lake
01h 25mBest practices and solution design using data versioning for machine learning
33mBuilding Reproducible ML Processes with an Open Source Stack
54mThe Future of Data Science/ML at Scale: A Look at MLflow, Delta Lake + Lakehouse
26mCI/CD For Data – Building Data Development Environments With Open Source Stacks