Data Mastering at Scale with AI
summary
Enterprise MDM replaces brittle fuzzy keys with a pipeline that uses active learning, clustering and classification to learn blocking and matching rules, cutting quadratic comparisons. Training needs are small — about 100–150 labeled pairs for models that scale to millions — and production results range roughly 80–98% accuracy depending on data quality. Implementation ties models into ETL tools like StreamSets/Kafka and runs on Spark + graph DB for merging, governance and downstream publishing.
AI-written summary · about
chapters (10)
- 0:00Introduction and background
- 1:01Why data mastering matters
- 7:10Data silos and their impact
- 11:21Challenges in traditional MDM
- 16:25AI's role in data mastering
- 21:47AI techniques for entity resolution
- 23:37Q&A: AI algorithms for entity resolution
- 26:12Q&A: Active learning in practice
- 30:53Q&A: Architecture and resources
- 35:19Closing and next steps
related talks
24mHow Graph-based Master Data Management Accelerates Data-powered Projects
21mData Linkage Options: MDM vs Splink (PySpark)
24mSmart approach in development and deployment process for ML models
08mUsing machine learning to infuse data operations
27mScalable Data Pipelines - an ML Approach
25mModernizing Your Data Strategy With Machine Learning & The Cloud
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications
34mScaling ML in the Traditional Enterprise