1 + 1 = 1 or Record Deduplication with Python
summary
Uses a real 881-record dataset (about 150 true duplicates) and shows Python code that cleans names, geocodes addresses with geocoder / Google geocoder, and uses the recordlinkage toolkit to compare fields with Jaro‑Winkler and a geographic exponential decay. Blocking by postal code cuts ~300,000 potential pairs to ~6,000 candidate pairs. Compares simple weighted threshold, an SVM (133 TP, 2 FP, 17 FN), and Dedupe/active learning (141 TP, 3 FP, 9 FN), then applies clustering to form transitive duplicate groups.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker
- 1:05Motivation for deduplication
- 5:24String similarity algorithms
- 7:30Data preprocessing steps
- 9:16Indexing with blocking
- 10:43Comparison vectors
- 13:38Threshold classification
- 16:31Supervised classification
- 19:01Active learning approach
- 21:52Clustering matches
- 24:45Q&A and resources
related talks
24mMachine learning techniques for data cleaning
38mScalable Patient Records De duplication using machine learning
40mData Deduplication using Locality Sensitive Hashing
35mBeyond Spark MLlib: Deduplicating Common Crawl at Scale
33mFuzzy Search Algorithms How and When to Use Them
46mData needs testing too
52mUsing Python to Get Out the Vote
36mNear-Duplicate Ad Detection in Online Classified Ad Services