Almost Perfect: A Benchmark on Algorithms for Quasi-Experiments
summary
Compares Difference‑in‑Differences, Synthetic Control, Meta‑Learners and Graphical Causal Models on three open datasets with 200 simulations per dataset, reporting bias, RMSE and interval coverage. Finds no universal winner: DiD is stable but not best, S‑learner/T‑learner/double‑robust share similar bias while double-robust shows better coverage, unconstrained synthetic control can catastrophically extrapolate, combining models gives limited gain, and using user‑level features materially improves accuracy—granular data cuts RMSE ~30%.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:08Speaker background
- 2:53Quasi-experiments overview
- 4:58Difference-in-difference & synthetic control
- 7:04Metal learners: S, T, double robust
- 9:34Graph causal models & assumptions
- 10:37Benchmark setup & data
- 12:07Model bias in simulations
- 16:25Confidence intervals & coverage
- 19:41Key insights from results
- 21:05Improving models with ensembles
- 25:41Tuning results & recommendations
- 26:44Takeaways & limitations
- 27:49Q&A
related talks
11mCausal Inference For Socio-Economic & Engineering Systems
46mWhat's the Science in Data Science?
41mKeynote: Personalization, Bandits, and Causal inference
01h 19mModel criticism with ArviZ
30mMachine Learning Techniques for Class Imbalances & Adversaries
15mInterpretable Machine Learning with R
26mParadoxes in Data Science
33mThe Challenge of Valid Evaluations