Open Source Entity Resolution Using Zingg
summary
Zingg is an open source entity resolution system that combines a learned blocking model with a pairwise classifier, graph clustering and active learning to avoid hard‑coded rules. It runs distributed on Spark and has a Snowflake variant, adds cluster IDs with min/max match scores, and claims practical scale — a fashion retailer resolves about 80 million customer profiles per day. The system produces micro‑buckets, trains from a few dozen labels, and outputs merged cluster identifiers.
AI-written summary · about
chapters (18)
- 0:00Zingg and entity resolution
- 1:58Founder's journey
- 6:39Defining identity
- 9:10Rule-based resolution challenges
- 10:54Design goals for Zingg
- 13:22Machine learning in Zingg
- 16:34Blocking and similarity models
- 19:00Training data and active learning
- 20:46Zingg output and deployment
- 22:55Zingg in data stack
- 24:23Gift card fraud prevention use case
- 26:28NC campaign funds transparency
- 28:18Identity in enterprise data
- 29:19Call to action
- 30:08Q&A: Reducing comparisons
- 31:55Q&A: Field weighting
- 33:43Q&A: Academic research
- 35:09Closing remarks
related talks
41mOpen Source Entity Resolution - Needs and Challenges
29mA Graph Entity Resolution Playbook
26mBuilding an Entity-Resolved Knowledge Graph: Why and How
38mZGo: Year One
37mMachine Learning models utilization
28mCombining Third-Party Data, Entity Resolution, and Graph Analytics for ESG Risk Mgmt
28mCreating Enterprise Knowledge Graphs from Unstructured Data
32mHumans In the Loop: Collaborating in the Age of AI