The Hitchhiker's Guide to Advanced A/B Testing Techniques
summary
Covers concrete checks and fixes: run a sample ratio mismatch test and stop if allocation deviates (planned 50k users, saw ~20k so far, initial split 10k vs 9.5k flagged), trace segment/time/config bugs (personalization algorithm skipped low-data users) and reassign them to control, which cuts observed lift from 16% to 10% but keeps p < 0.05. Use sequential stopping with boundaries to avoid sequential (peeking) inflation to 20–30%, and apply CUPED reduces variance up to 80% to shorten runtimes 20–70%.
AI-written summary · about
chapters (10)
- 0:00Introduction to A/B testing
- 1:09Why A/B tests work better
- 4:47Personalized notifications demo
- 7:20Early analysis pitfalls
- 8:25Fixing sample ratio mismatch
- 14:56Peeking at results too soon
- 19:17Sequential testing benefits
- 21:45CUPED method explained
- 25:00Running parallel experiments
- 27:11Summary and key takeaways
related talks
27mHow to A/B test with confidence
26mAnti-Patterns in A/B Testing - or How does AB Testing Scale?
18mHow Data Scientists Broke A/B Testing (And How We Can Fix It)
29mBalancing Personalization & A/B Testing in a Customized World
15mA/B Testing: Test Your Own Hypotheses & Prepare to be Wrong
09mA/B tests: Understanding user behaviour
34mA/B Testing You'll Like
24mThe Scientific Method of Experimentation