Model validation using deep generation of stress data
summary
Uses a weekly US gasoline dataset to stress-test a forecasting pipeline: a LSTM baseline (input size 10 weeks, horizon 100 weeks, backtesting error ≈0.24–0.25) is evaluated for worst‑case risk by splitting features into mutable vs immutable and fitting an expectation model plus a quantile model. A GAN with TCN generator/discriminator is trained on the test split to synthesize 5× more samples, and GenRisk-style evaluation shows similar mean error but narrower confidence intervals on risk estimates when using generated stress data.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:33Data set shifts in models
- 4:30Concept vs covariance shift
- 10:17Model robustness with worst case risk
- 14:28LSTM forecasting setup
- 20:04Mutable vs immutable variables
- 33:49Synthetic time series generation
- 44:42Generative model quality
- 50:07Fake data for stress testing
- 56:43Generative stress testing challenges
- 59:36Summary
related talks
27mSynthetic Data in Financial Time Series
45mForecasting at Scale
27mModel Evaluation for Humans
31mML time series forecasting methods
01h 28mFeature Engineering Made Simple
26mTesting a Data Science Model
09mTesting Strategies for Generative AI Applications
29mGANs: The Case of Predictive Business Process