Usable A/B testing – A Bayesian approach
summary
Bayesian A/B testing uses conjugate posterior beta distributions and Monte Carlo sampling (numpy or PyMC) to report probabilities and credible intervals instead of p values. An experiment at ResearchGate yields modes of 53.9% and 59% and a result of 85% chance of >8% lift and ≥6.4% guaranteed lift from the sampled posterior (about 1,000,000 Monte Carlo samples). The workflow supports sequential stopping rules and a reusable evaluation library that is being released.
AI-written summary · about
chapters (13)
- 0:00Bayesian A/B testing intro
- 1:45A/B experiment considerations
- 6:45Hypothesis testing problems
- 14:42Why ResearchGate chose Bayes
- 16:27Bayes’ theorem explained
- 19:43Beta distribution and priors
- 22:23Analyzing A/B test results
- 27:44Sequential testing criteria
- 30:56Business decisions with Bayes
- 31:16Q&A: Non-binary metrics
- 32:58Q&A: Code examples
- 33:20Q&A: Prior selection
- 36:34Q&A: Visual Website Optimizer
related talks
39mSuccessfully Applying Bayesian Statistics to A/B Testing in Your Business
25mScaling Bayesian Experimentation
41mA/B Testing with Style
34mHypothesis to Impact: Mastering the Art and Science of Controlled Experiments
33mTwo Years of Bayesian Bandits for E Commerce
47mexpectation: A modern take on statistical A/B testing with e-values and martingales
01h 30mBayBE: A Bayesian Back End for Experimental Planning in the Low-To-No-Data Regime
29mHow to use Data Science Superpowers in real life, a Bayesian perspective