Data Linkage Options: MDM vs Splink (PySpark)

on YouTube ↗21m

" summary

Direct comparison of Master Data Management vs Splink: MDM provides a non‑technical UI, integrated QA/distribution and predictable outcomes but typically costs £75–150k/yr (with Customer Insights at ~£7k/yr) and uses rule/fuzzy matchers; Splink is a free Python library built by the Ministry of Justice, scales to 100+ million records, uses probabilistic matching with blocking, runs on DuckDB or Spark/Databricks/Synapse, returns match weights (needs technical build and human review) and pairs well as a heavy-duty matcher feeding MDM.

AI-written summary · about

" chapters (7)

1. 0:00Introduction to the challenge
2. 1:32Understanding record linkage
3. 2:31Master data management (MDM)
4. 6:40Introduction to Splink
5. 10:28Splink's matching methodology
6. 14:25Splink performance and pros/cons
7. 16:30Conclusion and Q&A

" related talks

• 37mData Mastering at Scale with AISonal Goyal · Data Con LA 2020
• 24mHow Graph-based Master Data Management Accelerates Data-powered ProjectsTim Ward · Big Data LDN 2022
• 24mSmart approach in development and deployment process for ML modelsDanijel Ilievski · DSC Europe 2022
• 50mApache Spark vs Machine LearningPatryk Pilarski · Confitura 2019
• 36mBig Data Modeling with Spark SQL: Make data valuableJayesh Patel · Data Con LA 2019
• 21mCase Law and ML on SparkJeremy Corbett · Data By the Bay 2016
• 24mDataEngBytes 2023 - MEL-T3-03 - Akira Wang & Abdi FarahAkira Wang, Abdi Farah · DataEngBytes Melbourne 2023
• 38mEnabling Big Data and Machine Learning for the Masses...Tim Kral · Scale By The Bay 2018