Evaluating the Unseen: Supervised Evaluation for Unsupervised Algorithm
summary
Uses a supervised edge classifier to label account pairs so clustering quality can be measured with v-measure instead of relying on silhouette scores. Features include merchant overlap, account age, purchase frequency, product overlap, hierarchical address match and return behavior. The classifier yields about 0.95 PR-AUC on held-out labels, handles complex cases with ~90% connection rate, and lets the team compare the online graph clustering to a baseline graph solution (~94% precision). Operates at scale on ~4.8M transactions/day with 300 ms latency targets.
AI-written summary · about
chapters (9)
- 0:00Introduction and background
- 2:34Identity resolution problem
- 8:04Clustering and graph basics
- 11:02Unlabeled evaluation methods
- 12:55Supervised evaluation approach
- 17:21Modeling expert labeling process
- 21:42Classification model results
- 23:27Advantages of edge model
- 24:57Summary and takeaways
related talks
25mUnsupervised shallow learning for fraud detection on marketplaces
47mAssessing the quality of a clustering
18mAnti-Money Laundering ML Modelling Approach
28mHow Machine Learning Can Achieve Data Quality Monitoring At Scale
29mEvaluation Design
06mUnifying Online and Offline Worlds
31mLet's do graph machine learning on Cloud
40mMachine Learning Model Evaluation Metrics