Ted Dunning, Chief Application Architect, MapR
summary
Describes extending co‑occurrence recommenders to near real‑time updates by pruning histories with an interaction cut and capping item frequency (frequency cut), turning a quadratic self‑join into linear work and bounded per‑event cost (≈2k−1 operations, amortized O(1)). Implements co‑occurrence → significant indicators pushed into a search engine for delivery, uses Hadoop/Spark/Mahout-era tooling and MapR infrastructure, shows empirical wins (handled ~250 billion observations historically; million‑song data drops ~6B pairs to ~600M, ~10× reduction) and that many updates are prunable so the online system works.
AI-written summary · about
chapters (10)
- 0:00Start
- 5:21Problems with Raw Co-occurrence
- 5:41Overview: Get Useful Indicators from Behaviors
- 5:52Which one is the anomalous Co-occurrence?
- 8:21Collection of Documents: Insert Meta-Data
- 9:23Cooccurrence Mechanics
- 10:32Cross-occurrence Mechanics
- 16:51Batch Scaling in Time Implies Scaling in Space
- 19:10Real-time recommendations using Map data platform
- 22:11Recap
related talks
27mTed Dunning, Chief Application Architect, MapR
52mDevil in the Haystack
20mDataEngBytes 2023 - PER-T1-06 - Kamal Hossain
31mCreating recommender systems with Spark, Scala and Prediction.Io
39mRecommendations at XING: From theory to production
42mScaling to Billions of Real-time Recommendations Daily
56mSpark-driven Audience counting
37mSearch and Recommendations: 3 Sides of the Same Coin #bbuzz