Time Travel and Provenance for Machine Learning Pipelines
summary
Claims that intrusive code hooks are replaced by standardized change‑data‑capture from the distributed file system, feature store, resource manager and apps, implemented in Hopsworks and pushed through an e‑pipe into a provenance store; uses Hoodie deltas for time‑travelable feature versions and HDFS xattrs plus application IDs and Airflow job context to reconstruct relations. Conventions, path‑ and tag‑based filtering and operation coalescing cut thousands of file events to a manageable graph, enabling reproducible debugging, automated pipeline reruns and garbage‑collection decisions in practice.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:21Abstract
- 2:17Track Provenance
- 3:34Why Provenance
- 6:54System Challenges
- 7:49Consistency
- 8:20Transparency
- 9:02Number of Operations
- 9:34File System Operations
- 10:57Application and Framework Libraries
- 12:33Conventions
- 13:21Pathbased Filtering
- 14:09Tagbased Filtering
- 15:05Enable Provenance
- 16:39ML Pipeline Components
- 17:06Why is it important
- 18:57Conclusion
related talks
43mData Pipelines à la Mode
39mEnd to End Machine learning pipelines for Python driven organizations
30mSMART DATA PIPELINES FOR AI
28mData Validation for Feature pipelines: Using Great Expectations and Hopsworks
23mHopsworks Feature Store after 4 years: Lessons learned and what's next
40mCreating and operating ML models from event based data using feature stores
30mDesigning Resilient Data Pipelines
27mThe Sanity Check: Observability for AI Data Pipelines