Anti-Patterns in A/B Testing - or How does AB Testing Scale?
summary
A/B testing fails when it is treated as a single library, when every visitor is assigned to every experiment, or when analysts get dumped a raw data pile with no entrance timestamps; robust systems log when a user actually sees the control or treatment, coordinate web mobile and server assignments to avoid divergent libraries, and automate analysis (Python pipelines plus Bayesian methods) to keep a history of experiments; aim for one A/B test every two weeks per team and accessible runbooks so tests aren’t rerun unknowingly.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:42What is A/B testing
- 7:09Common A/B testing antipatterns
- 14:48A/B testing culture
- 19:54Python in A/B testing
- 21:00Success in A/B testing
- 26:01Q&A
related talks
27mHow to A/B test with confidence
15mA/B Testing: Test Your Own Hypotheses & Prepare to be Wrong
28mThe Hitchhiker's Guide to Advanced A/B Testing Techniques
18mHow Data Scientists Broke A/B Testing (And How We Can Fix It)
30mStats Don't Have to Be Scary: Automatic a/b Test Analysis Using Python
29mBalancing Personalization & A/B Testing in a Customized World
34mA/B Testing You'll Like
09mA/B tests: Understanding user behaviour