From research to production at ebay
summary
eBay clusters titles from hundreds of millions of items (uploader says ~400M, talk cites ~800M) into a similarity graph using MinHash LSH to avoid 10^16 comparisons, then computes connected components for product clusters and also emits top‑K nearest neighbors. Implementation uses Scalding -> Spark with Luigi pipelines, Bloom filters and count‑sketches to shrink dictionaries, and runs at scale (hundreds of millions of vertices) but hits production issues: YARN memory contention, oversized buckets, and a pragmatic fix of emitting top‑K and discarding noisy buckets.
AI-written summary · about
chapters (17)
- 0:00eBay research project intro
- 1:51Research problem defined
- 3:38Clustering 800M items
- 5:44Feature extraction from items
- 9:44Scaling with MinHash and graphs
- 13:15Spark for connected components
- 16:04Scalding vs MapReduce
- 21:21Algebird for efficient processing
- 23:32Bloom filters for joins
- 25:30Research to production at eBay
- 27:17Production challenges
- 31:28Spark on YARN struggles
- 34:38Refining merchandising solution
- 36:24Handling item similarity edge cases
- 38:50Debugging production hurdles
- 41:58Spark memory constraints
- 43:28Q&A on components and LSH
related talks
45mTransformer based clustering: Identifying product clusters f...
36mSearch Complete
18mEhtsham Elahi, Senior Research Engineer, Netflix @ MLconf SEA
27mData @scale Seattle - Yucheng Low
26mLarge Scale Graph Processing & Machine Learning Algorithms
22mHarnessing image and text data for product matching
35mBeyond Spark MLlib: Deduplicating Common Crawl at Scale
44mFeature store