Realtime Bayesian A-B testing with Spark Streaming
summary
Realtime pipeline prototype uses Spark Streaming with Kafka to feed live experiment data into a Bayesian engine that models conversions with a beta-binomial model and samples posteriors to compute Bayes factors and empirical Bayes priors. Claims include safe sequential stopping rules (e.g. stop when Bayes factor > 9) that keep false discovery low, examples on millions of samples, 20–30s lag in the pipeline, and working prototypes with real Booking.com experiments.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:14Why we explored Bayesian A-B testing
- 10:50A-B testing basics
- 13:19Bayesian vs frequentist methods
- 18:27Beta-binomial model for conversions
- 21:44Highest density interval and ROPE
- 24:57Frequentist sequential testing risks
- 26:24Bayes factors for stopping rules
- 28:54Empirical Bayes for prior estimation
- 30:20When sequential testing fits
- 31:44Real-world results
- 35:16Q&A
related talks
25mScaling Bayesian Experimentation
04mInterview
16mImproving Streaming Experience with Bayesian Optimization, from AB to...
45mReal-time, Streaming Advanced Analytics....
29mBatch and Streaming Processing in the World of Data Engineering and Data Science
18mReal-time Experiment Analytics at Pinterest with Apache Flink
39mReal-time Transaction Categorization w/ Bayesian Feedback Loop
30mSmart Streams: A Real-time framework for scoring Big and Fast Data