How to A/B test with confidence
summary
Recap of concrete rules and gotchas: compute sample size and power (if calculator says a million but there are only a thousand users the test is infeasible), allocate users as late as possible (100k visitors but only 10k reach checkout so split at checkout yields 50% vs 52.7% not 5% vs 5.2%), never roll your own randomization — prefer vanity style hashing of userID+experiment, beware implementation bugs that crash a variant 10x, and know platform quirks like Google Optimize client-side flashing and practice with an AAA test.
AI-written summary · about
chapters (15)
- 0:00Start
- 0:50The Plan
- 2:06Not just buttons!
- 5:31Sample size
- 7:09Bayesian A/B testing
- 11:08Non random split
- 14:52Not agreeing setup
- 18:38Repeated significance testing
- 20:03Do the maths
- 27:27If the result is 'wrong'
- 28:05Don't reinvent the wheel
- 29:18Resist the urge to check/tinker
- 30:20Be wary of overtesting
- 31:30Document your test
- 32:26Further Reading
related talks
28mOptimizing your A/B testing roadmap
38mTesting, Testing: On Experimental Drift and Data Driven Product Design
34mHypothesis to Impact: Mastering the Art and Science of Controlled Experiments
09mDemystifying the Dreaded A/B Test
18mData Science Careers via A/B Testing
45mHow we killed 80% of features and increased outcomes of a/b testing by 100%
32mAB Testing to Determine Product Priority
36mHidden engineering challenges of A/B Testing