Efficient Time-Series Forecasting with Thousands of Local Models on Databricks
summary
Shows scaling local models on Databricks using Pandas API on Spark, training thousands of per‑group models in parallel; backtesting runs 20,000 parallel folds that yield about 25% accuracy improvement in 23 minutes; production pipeline generates over 1 million forecasts; explores storing binaries in delta and tradeoffs with MLflow API limits and Unity Catalog, finds synchronous train‑and‑predict avoids storage bottlenecks and warns about debugging limits and subtle leakage risks.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:05Business needs in utilities
- 3:15Granular forecasting challenges
- 4:46Scaling models with Pandas on Spark
- 6:57Synchronous training workflow
- 8:42Asynchronous training approaches
- 12:39Backtesting for evaluation
- 15:35Production impact metrics
- 17:00Lessons from scaling models
- 18:30Future of scaling frameworks
- 19:18Q&A: Price volatility
- 21:14Q&A: Local vs pooled models
- 23:01Q&A: Weather forecast errors
- 24:25Q&A: Local and global models
- 25:33Q&A: Small dataset training
- 26:37Q&A: Production scaling challenges
related talks
26m“Many-model” Time Series Forecasting: Scaling PyTorch Training Across 1000s...
33mDealing with the versioning of production-ready models
28mScaling Machine Learning Training and Inference
29mScaling to Millions of ML Models to Solve the Problems of SRE and Security
40mMulti-Series Forecasting at Scale with StatsForecast
42mRobust Automated Forecasting in Python and R
37mForecasting using machine learning methods
45mForecasting at Scale