How Data Scientists Broke A/B Testing (And How We Can Fix It)
summary
A/B testing practices cause misalignment by optimizing for statistical guarantees instead of business decisions; the talk argues for using non-inferiority margin tests (e.g., power tests keyed to an acceptable loss like ~1.5% conversion) and computing the expected value of sample information via prior-driven simulations to decide when extra weeks of data are worth the opportunity cost. Implements these ideas with R and Shiny, simulating priors/posteriors, estimating expected value of two-week extensions, and building an analytics engine that product teams use to trade off time, risk, and dollars.
AI-written summary · about
chapters (6)
- 0:00The AB testing dilemma
- 2:29Bridging the business gap
- 5:30Non-inferiority testing
- 7:13Value of information
- 13:36Reframing data science
- 17:14Q&A and conclusion
related talks
27mHow to A/B test with confidence
01h 03mExperimentation in Data Science
26mAnti-Patterns in A/B Testing - or How does AB Testing Scale?
15mA/B Testing: Test Your Own Hypotheses & Prepare to be Wrong
28mThe Hitchhiker's Guide to Advanced A/B Testing Techniques
29mBalancing Personalization & A/B Testing in a Customized World
24mThe Scientific Method of Experimentation
29mIntuitive A/B Test Evaluations for Coders