Building your own Distributed Streaming Recommendation Engine

on YouTube ↗40m

" summary

A distributed streaming ALS implemented with stochastic gradient descent updates per incoming rating, using Apache RDDs and Apache Spark DStreams and Kafka for 5s windows (1,000 records per mini‑batch). Model stores user/item latent vectors plus global and per‑entity biases, initializes unseen IDs with random vectors and uses full‑outer joins to merge new and existing factors. Evaluation on MovieLens (~26M ratings) shows streaming RMSE trends toward batch ALS but highlights cold‑start, hyperparameter search cost, and shuffle/lookup performance issues. MovieLens 26M

AI-written summary · about

" chapters (39)

1. 0:00Introduction
2. 0:43Recommendation engines
3. 1:40Collaborative filtering
4. 3:16Factorization
5. 4:49Derivation
6. 5:54Regional Ratings Matrix
7. 6:12Demo
8. 6:52Assigning Ratings
9. 7:02Ratings Matrix
10. 8:05Rating Factorization in Streaming
11. 9:10Stochastic Gradient Descent SGD
12. 9:33Biases
13. 10:44Stochastic gradient descent
14. 11:31The practical difference
15. 13:48RDDs
16. 14:20Spark Sommelier
17. 16:10Streaming Recommendation
18. 17:28MLE API
19. 18:12Factor class
20. 18:36Summary
21. 19:20Special Scenario
22. 19:57Limit Case
23. 21:16Random Feature Vector
24. 21:42User and Product Weighting Factors
25. 22:37Product Weighting Factors
26. 23:00Global Bias
27. 23:46Predicted Rating
28. 24:13Getting the Error
29. 25:00radiance
30. 26:21new observations
31. 26:43weighting factors
32. 27:43data set
33. 29:07training batch LS model
34. 29:40validation RDD
35. 31:35results
36. 32:33problems
37. 34:46streaming version
38. 35:12grid search
39. 35:57performance considerations

" related talks

• 01h 48mRecommendation engines explainedSophie Watson, Rui Vieira · DevConf.CZ 2019
• 37mBuild a Recommender System in Apache Spark and Integrate it Using AkkaWillem Meints · Scala Days Berlin 2016
• 45mReal-time, Streaming Advanced Analytics....Chris Fregly · Data By the Bay 2016
• 32mBuilding an recommendation system for IPTVTomislav Hlupic, Iva Soric · Data Science Conference 4.0
• 48mBuild a Recommender System in Apache Spark and Integrate It Using AkkaWillem Meints · Scala Days Berlin 2016
• 40mIntroduction to Machine LearningJames Ward · Scale By the Bay 2017
• 41mRecommendations for Building Machine Learning SoftwareJustin Basilico · Data By the Bay 2016
• 34mActionable Insights with Real-time Streaming Analytics of Customer ReviewsDaniel Wrigley · Big Data Conference Europe 2019