Maximizing marketplace experimentation: switchback design for small samples
summary
Conventional AB testing fails when units are few and effects are tiny so switchback design alternates treatment periods to cut interference and autocorrelation and to enable more within-unit comparisons. Claims: real projects report >2x precision and occasionally ~4x, turning a four month run into one. Worked tools: period length optimization, switch frequency tuning, and flight‑level case studies in airlines; used by Uber Lyft Doordash. Watch for 1% uplift detection limits, temporal carryover, and costly frequent switching tradeoffs.
AI-written summary · about
chapters (15)
- 0:00AC Metric and speakers
- 1:05Why experiments fail
- 4:58Limitations of A/B testing
- 7:28Small samples and subtle effects
- 11:47Switchback design
- 14:38Switchback for two-sided markets
- 17:33Switchback precision increase
- 21:13Reducing autocorrelation
- 22:15Implementing Switchback
- 25:50Switchback real-world impact
- 27:17Q&A on Switchback
- 28:23Handling external influences
- 30:08Adopting Switchback
- 31:13Robust Switchback experiments
- 33:20Closing remarks
related talks
44mA B Testing: Harder than just a color change
41mA/B Testing with Style
19mPricing, Supply, A/B and Switchback Testing in FoodTech
01h 03mExperimentation in Data Science
20mWinning the Game: Leveraging Contextual Bandits for Creative Optimization
30mStats Don't Have to Be Scary: Automatic a/b Test Analysis Using Python
13mProduct Experimentation for Growth
35mThe Realities of A/B Testing