1 + 1 = 1 or Record Deduplication with Python
summary
Record deduplication uses concrete signals: Python code that applies fingerprinting and blocking (unidecode for accents, token ordering, geocoding via Google and haversine for address distance), then scores pairs with similarity functions and a classifier (random forest via the dedupe-style workflow and active learning to collect labels). Small examples show 6 records produce 15 pairs; at scale one million records yields ~500 billion pairwise checks unless blocked. Real impact: Census deduping finds 250,000 fewer people.
AI-written summary · about
chapters (5)
- 0:00Start
- 1:58Fusee Similarity and Data Cleaning
- 7:37Fingerprinting and Blocking
- 27:14Privacy Implications of Record Linkage
- 28:20References
related talks
39mRecord linkage: Join for real life
38mKeynote: Using the Python Data Science Stack to Determine
27mRapid deduplication and fuzzy matching of large datasets using Splink
26mNo, Maybe and Close Enough: Using Probabilistic Data Structures in Python
31mHoly D@t*! How to Deal with Imperfect, Unclean Datasets
41mSimilar entity detection in large data
26mNo, Maybe and Close Enough! Probabilistic Data Structures with Python and Redis
12mThe Dilemma between Deduplication and Locality: Can Both be Achieved?