Backtesting and error metrics for modern time series forecasting
summary
Backtesting is a tradeoff: choose rolling vs expanding window, step size and refitting frequency to balance reliable error estimates against time and memory. Use larger step sizes and less frequent refits to save compute; almost never move one step at a time. Prefer pooled scale‑dependent metrics like normalized RMSE or scale‑independent ones (MASE, WMAPE) depending on whether scale matters. Hierarchical coherence requires squared‑error objectives. Use skforecast or Funk time for robust, efficient backtesting.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:29Modern time series forecasting
- 4:18Backtesting fundamentals
- 6:44Choosing backtest parameters
- 9:52Refitting frequency
- 11:57Edge cases in backtesting
- 13:43Backtesting libraries
- 14:04Error metrics structure
- 16:57Flaws in error metrics
- 18:22Key error metric properties
- 20:33Selecting metrics for series
- 24:08Scale-independent metrics
- 28:00Hierarchical forecasting
- 30:30Metrics for intermittent series
- 31:33Conclusion
related talks
32mAdopting Open-Source Tools for Time Series Forecasting
27mBenchmarking Time Series Foundation Models with sktime
43mFeature Engineering for Time Series Forecasting
17mCodifying Data Science Intuition
30mMy forecast is better than yours! What does that even mean?
39mForecasting With Classical and Machine Learning Methods
42mRobust Automated Forecasting in Python and R
34m5 ways to fail with time series