Scalable Data Pipelines - an ML Approach
summary
Argues that ML can cut months of manual schema mapping and the 60–70% of analyst time spent on cleaning by replacing rulebooks with column word vectors for schema mapping, using weak supervision (Snorkel) to turn noisy heuristics into training labels, and active learning selects ~500 edge cases instead of labeling 100k+. Solves scale by blocking via clustering to avoid n(n−1)/2 comparisons (10k records → ~50M pairwise checks; 1M → trillions), and applies NLP for unstructured sources and automated transform suggestions.
AI-written summary · about
chapters (5)
- 0:00Introduction to talk
- 1:32Data integration and mastering
- 8:39Challenges in data integration
- 15:34Machine learning for data pipelines
- 26:16Q&A
related talks
18mDesigning a scalable ML Ops Pipeline: Insights and best practices
18mAutomate the boring ML stuff with pipelines
39mMachine Learning with Scala on Spark
33mBuilding Resilient (ML) Pipelines for MLOps
24mSmart approach in development and deployment process for ML models
17mCrafting Superior ML Pipelines with SapientML
10mML-based Log Analysis for Faster Debugging
41mML plumbing