Productionising Data Science at Scale
summary
Tesco describes production systems that run tens of thousands of forecasts weekly, scale to ~3,000 stores and use Python UDF per row on the cluster with PySpark as a distribution wrapper; forecasting uses rolling-horizon scoring (MAPE on the last year), test‑driven development, git and code review, and an early warning system of automated stability tests. Recommendation work separates candidate generation and ranking (recall-focused top 300 candidates) with real‑time features, 10 million daily users and 2,000 hourly sessions feeding the pipeline.
AI-written summary · about
chapters (25)
- 0:00Tesco data science team intro
- 1:29Team structure and business impact
- 2:13Forecasting challenges
- 5:12Production forecasting needs
- 7:41Measuring forecasting success
- 9:05Production-ready data science
- 10:33Iterative forecasting design
- 12:18Scaling to thousands of stores
- 15:05Multi-level forecasting pipeline
- 16:09Early warning systems
- 17:59Judgmental overlays
- 19:22Data storage and improvement
- 20:25Cross-functional forecasting
- 21:06Recommendation systems intro
- 22:11Big data for recommendations
- 23:56Low-latency API serving
- 25:01Training-serving sync
- 25:41Recommendation system goals
- 27:07Generic recommendation framework
- 28:10Two-layer recommendation framework
- 30:17Basket recommendation example
- 31:40Feature engineering for ranking
- 33:03Supervised learning for ranking
- 34:27Closing remarks
- 34:50Question and answer session
related talks
28mHow Human Expertise and AI drive Scalable Forecasting Success at
18mUsing Transformer Models to Predict Demand at Scale in Production
23mMachine Learning with a Deep Dive into Automated Forecasting System
42mHow mobile.de brings Data Science to Production...
28mDemand Forecasting: Accuracy, Stability, Automation
01h 23mA Hands-On Introduction to Your First Data Science Project
29mProbabilistic demand forecasting at scale
28mEfficient Time-Series Forecasting with Thousands of Local Models on Databricks