Scaling Bayesian Experimentation
summary
Argues that Bayesian A/B testing uses priors on sample means (normal + inverse-gamma) so posteriors have closed form and avoid slow MCMC, and shows a production pipeline that scales bootstrapping with Spark by compressing data into frequency tables and sampling from cumulative frequencies to make millions of draws fit in memory, runs as daily batch processing, and proposes an incremental update using binomial draws to adjust bootstraps for newly arriving users rather than recomputing everything.
AI-written summary · about
chapters (9)
- 0:00Introduction to Bayesian experimentation
- 1:44Why Bayesian over frequentist
- 3:08What are priors in Bayesian stats
- 6:40Generalizing priors for all metrics
- 9:33Is Bayesian experimentation fast enough
- 12:04Bootstrapping explained and optimized
- 16:21Compressing data for bootstrapping
- 20:19Real-time updates to bootstraps
- 23:50Recap and open source resources
related talks
37mRealtime Bayesian A-B testing with Spark Streaming
38mUsable A/B testing – A Bayesian approach
39mSuccessfully Applying Bayesian Statistics to A/B Testing in Your Business
41mA/B Testing with Style
01h 30mBayBE: A Bayesian Back End for Experimental Planning in the Low-To-No-Data Regime
42mBayes and Bool Walk into a Bar
16mImproving Streaming Experience with Bayesian Optimization, from AB to...
31mCost-effective data annotation with Bayesian experimental design