Similar entity detection in large data
summary
Applies a multilayer perceptron on per‑pair feature distances (first/last name, address, DOB, gender), using custom distance functions and Levenshtein‑style string distance; training set is tiny (under 100 labelled pairs) and achieved ~89% recall but only ~65% precision. Pairwise scoring is split with MapReduce and run as overnight batch with incremental addition of new records, plus a UI feedback loop for relabeling; result ships to a customer for parallel evaluation but is not yet in production. Rule‑based baselines and random undersampling are discussed as next steps.
AI-written summary · about
chapters (26)
- 0:00Start
- 0:49Presentation Coverage
- 1:29Machine Learning for the masses
- 2:37Organization Data - Top level view
- 3:06Organization Data - Individual view
- 3:42Example Data for 3 persons
- 4:47Scenario 1 . Fraud Detection in Insurance Claims
- 6:11Similar Entity Detection Challenges
- 8:16Sample rules in Traditional approach
- 8:44Traditional Approach - Limitations
- 9:53Solution
- 10:39Types of Machine Learning algorithms
- 12:37Choice of Approach and Algorithms
- 15:44Metrics to validate the model
- 16:26Example detailing the metrics
- 17:32Supervised Learning Flow
- 19:55Algorithm Chosen
- 20:19Retrieve values and calculate distances
- 21:33Training the neural network model
- 22:58Model Validation
- 23:15Model Usage
- 23:51Our results
- 24:38Handling BIG DATA
- 24:57Large Data handling challenges
- 25:22Large Data handling ideas
- 26:05Way Forward
related talks
45mCustomizable probabilistic record linkage with Name Match
26mBuilding a Cross-Border KG: AI Powered Entity Resolution & Risk Detection
39mRecord linkage: Join for real life
19mIdentifying Personal Data by Fusing Elasticsearch with Neural Networks
37mSimilarity Search for Product Matching @ Semantics3
29m1 + 1 = 1 or Record Deduplication with Python
14mALERT: Machine Learning-Enhanced Risk Estimation for Databases Supporting...
15mOne in a billion: finding matching images in very large corpora