Robust Automated Forecasting in Python and R
summary
Automated pipeline makes millions of time‑series forecasts using Python and R and wins by specifics: 2.6 million forecasts in ~5.5 hours on 28 c4.8xlarge instances (1,008 cores), Docker + Celery on ECS with Luigi orchestration, Redshift input and S3 output, pragmatic preprocessing (log and Box‑Cox transforms, outlier handling), model pruning (drop pi‑flux and Python2), a 300s per‑model timeout that yields ~10% speedup, and cost cutting (use of spot instances gives roughly 40% cost reduction).
AI-written summary · about
chapters (14)
- 0:00Introduction
- 2:30Problem definition
- 4:36Time series preprocessing
- 9:59Outlier detection
- 10:21Forecasting models
- 12:34Forecast quality metrics
- 14:24Runtime optimization
- 23:48Python performance
- 25:14Hardware selection
- 32:06Pipeline finalization
- 32:28Docker scaling
- 35:11Architecture & concurrency
- 36:15Results & improvements
- 37:41Q&A
related talks
40mMulti-Series Forecasting at Scale with StatsForecast
15mMaking Complex R Forecast App...
29mSolving Forecasting Problems in R and Python
34mTime series for scikit-learn people
35mBuilding a successful inventory planning solution with Amazon Forecast
45mForecasting at Scale
28mEfficient Time-Series Forecasting with Thousands of Local Models on Databricks
29mProbabilistic demand forecasting at scale