Scalable Patient Records De duplication using machine learning
summary
An incremental, updatable model sits on a Python ML de‑duplication library and uses learned field weights, active learning and predicate blocking to avoid O(n^2) comparisons. The pipeline produces nightly updates, a cluster consolidation step and SQL-backed block generation. Measured results: an initial run over 2.35M → 233k patients in 34 minutes (≈2.1M duplicates), and an update matching 950k/47k new clusters in 24 minutes. Memory and consolidation of evolving clusters remain the main engineering tradeoffs.
AI-written summary · about
chapters (13)
- 0:00Patient claims duplication problem
- 3:14Exact match solutions fail
- 7:08Machine learning deduplication
- 9:36Deduplication Python library
- 12:28Training model with active learning
- 15:20Predicate blocks for complexity
- 20:28Solution constraints and requirements
- 22:59Step-by-step deduplication process
- 25:51Nightly updates and clusters
- 27:18Cluster consolidation for accuracy
- 30:30Performance and scalability results
- 34:24Key metrics from client data
- 36:33Future clustering improvements
related talks
35mBeyond Spark MLlib: Deduplicating Common Crawl at Scale
31mBig data for tiny patients: Applying ML to pediatrics
25m1 + 1 = 1 or Record Deduplication with Python
39mDuplicating Black Box Machine Learning Models
27mScalable Data Pipelines - an ML Approach
39mBest Practices for Prototyping ML Models for Healthcare
44mML in Production
20mAdvancing Epigenetic Predictors with Scalable Machine Learning