Intuitive A/B Test Evaluations for Coders
summary
Argues that resampling gives coders an intuitive path to A/B decisions by coding rather than advanced stats, using numpy random and SciPy to reproduce results similar to analytic tests. Uses a tiny example (about 12 users per group), runs 10,000 iterations of a shuffle permutation test to get p ≈ 0.025 (t test gives 0.0247), and builds a bootstrap confidence interval that excludes zero, so B looks reliably better.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:59Simple A/B scenarios
- 4:49Mean comparison limits
- 6:37Role of randomness
- 7:42Frequentist vs Bayesian
- 9:31Resampling methods
- 10:12Null hypothesis explained
- 13:02P-values with shuffling
- 15:56Shuffling vs t-tests
- 16:40Conversion rates evaluation
- 18:10Bootstrapping confidence
- 21:25Advantages and limitations of resampling
- 23:31Key takeaways
- 24:55Q&A: Gut feeling vs stats
- 26:40Q&A: Bayesian stats
- 28:02Q&A: Non-normal data
related talks
27mHow to A/B test with confidence
18mHow Data Scientists Broke A/B Testing (And How We Can Fix It)
28mThe Hitchhiker's Guide to Advanced A/B Testing Techniques
25mScaling Bayesian Experimentation
26mAnti-Patterns in A/B Testing - or How does AB Testing Scale?
34mA/B Testing You'll Like
29mBalancing Personalization & A/B Testing in a Customized World
09mA/B tests: Understanding user behaviour