Machine learning techniques for data cleaning
summary
Open-source Python libraries actually parse messy real-world fields: an address model uses conditional random fields to split tokens, probablepeople and parserator-style tools break names and product strings into labeled pieces, and a deduplication pipeline combines distance features with active learning. The dedupe library powers a hosted service, claims handles ~1 million records on a laptop, and reports 90% precision and 9% recall on test data; repositories are MIT licensed and include examples and a CLI.
AI-written summary · about
chapters (10)
- 0:00Data cleaning challenges
- 1:07Ambiguity in data cleaning
- 2:50ML for data cleaning
- 3:33US Address for parsing
- 8:19Probable People for names
- 9:26Custom parsers with Parser Aider
- 11:12Record linkage and deduplication
- 18:42DeeDee IO for non-programmers
- 20:28Open source tools
- 21:18Q&A on precision and recall
related talks
48mCleaning Messy Data at Scale: APIs, LLMs, and Custom NLP Pipelines
25m1 + 1 = 1 or Record Deduplication with Python
29mDirty Data Science Machine Learning On Non Curated Data
49m"Small Data" Machine Learning
29mBig Data Deduplication and data matching using Python
49mData Cleansing & Beyond: How to Address Data Debt for AI
54mMachine Learning explained with Examples
15mMachine learning: Necessity vs obligation