Prophet at scale to tune & forecast time series at Spotify
summary
Claims that distributed Prophet reduces runtime from days to about 30 minutes for ~10k series by running models on Spark on GCP with ephemeral clusters, costing roughly $5 per job. Details cover an in-house framework that parallelizes permutations for tuning, mixes brute‑force and Bayesian optimization, writes results to BigQuery/GCS, and uses Luigi/kubernetes for orchestration. Mentions concrete limits: 50 nodes ≈800 vCPUs, issues with many small output files and occasional random failures at very large scale. 10k time series
AI-written summary · about
chapters (7)
- 0:00Time series forecasting intro
- 1:08Spotify’s scale challenges
- 8:38Scaling Prophet with Spark
- 13:33Plug-and-play forecasting
- 20:58Parallel tuning with Bayesian opt
- 24:13Engineering adoption hurdles
- 31:40Q&A
related talks
37mForecasting Time Series Data at scale with the TICK stack
40mMulti-Series Forecasting at Scale with StatsForecast
28mEfficient Time-Series Forecasting with Thousands of Local Models on Databricks
06mLightning Talk: Predictive Autoscaling in Kubernetes With KEDA and Prophet
21mDon’t Call It 'The Forecast': Structured Prediction at Scale
45mForecasting at Scale
29mProbabilistic demand forecasting at scale
40mPowering Spotify's Audio Personalization Platform