Large-Scale Recommendation System with Python and Spark

on YouTube ↗26m

" summary

Kroger and 84.51 deploy an ensemble recommender that combines conditional filtering (co-purchase probabilities computed with PySpark on HDFS) with category-level logistic regression with L1 regularization (scikit‑learn) to predict where customers will try something new. The pipeline uses Airflow DAGs to schedule Spark jobs, weights category sampling, and pushes results via Kafka; it handles tens of millions of accounts, hundreds of recommendations per week and processes hundreds of billions of rows. An empirical surprise: 30–40% of basket items are new within 52 weeks.

AI-written summary · about

" chapters (23)

1. 0:00Intro
2. 1:01NOTES
3. 1:25CONTENTS
4. 1:57WHAT IS 84.51?
5. 2:54WHAT IS KROGER?
6. 3:24SETTING THE SCENE
7. 5:27KROGER'S (PERSONALIZED) DIGITAL PROPERTIES
8. 6:37TOOLSET
9. 8:10CONDITIONAL FILTERING OVERVIEW
10. 8:52CONDITIONAL FILTERING FUNDAMENTALS
11. 9:48CONDITIONAL FILTERING PYSPARK IMPLEMENTATION
12. 13:10CONDITIONAL FILTERING LIMITATIONS
13. 14:25CATEGORY TRIAL VIA MACHINE LEARNING
14. 15:15REGRESSION WITH L1/LASSO REGULARIZATION
15. 16:45REGRESSION EXAMPLE
16. 17:42ENSEMBLE PART 1 - VECTOR NORMALIZATION
17. 18:19VECTOR NORMALIZATION - EXAMPLE
18. 18:35ENSEMBLE PART 2 - WEIGHTED SAMPLING
19. 20:11APACHE AIRFLOW
20. 21:02DAG LAYOUT
21. 21:35SCHEDULING VIA PYTHON
22. 23:21DAGS CAN GET PRETTY WILD
23. 23:38INITIAL EXPERIENCE

" related talks

• 29mThe Challenges of Building a Personalized Recommendation System in E-CommerceP. Poznic · Data Science Conference 4.0
• 35mSpotting trends and tailoring recommendations: PySpark on Big Data in fashionMartina Pugliese · PyData Berlin 2016
• 37mBuild a Recommender System in Apache Spark and Integrate it Using AkkaWillem Meints · Scala Days Berlin 2016
• 31mCreating recommender systems with Spark, Scala and Prediction.IoFlorian Krause · code.talks 2017
• 28mThe Power of Personalized Customer Experiences Through AI InnovationsN. Trajkov · DSC Europe 23
• 32mLeveraging recommender systems to personalise search resultsSoraya Hausl · PyData London 2017
• 48mBuild a Recommender System in Apache Spark and Integrate It Using AkkaWillem Meints · Scala Days Berlin 2016
• 31mNLP text recommender system journey to automated training pipeline with Spark and SagemakerAditya Sakhuja · Scale By the Bay 2020