Versioning Multimodal Data: Metadata & Beyond
summary
The DVC team argues that multimodal needs a metadata-first versioning approach because files are heavy (megabytes to gigabytes) and datasets scale to thousands to millions to billions of files. A prototype called Data Chain serializes rich metadata with Pydantic models into a database, supports incremental updates (compute only changed items), exposes dataset history (nine demo versions) and enforces atomic dataset versions so sharing and queries work efficiently and the demo workflows succeed.
AI-written summary · about
chapters (9)
- 0:00Multimodal data versioning
- 2:49Metadata critical for multimodal data
- 9:19Heavy data with Python & databases
- 14:21Heavy data to big data solutions
- 15:03Complex data structures in databases
- 19:44Incremental updates for datasets
- 22:37Sharing & collaborating on datasets
- 24:03Conclusion and key takeaways
- 25:10Q&A
related talks
35mData versioning in machine learning projects
36mData Versioning - does it mean what we think it means?
25mManaging changes to your data with dat
29mMetadata Systems for End-to-End Data & Machine Learning
27mRevision Control for Structured Data
24mMake data central feature
43mVersioning, Syncing & Streaming Large Datasets Using DAT + Node
29mData Stored Everywhere and Nowhere - Building Applications on the Distributed Web