Stats Don't Have to Be Scary: Automatic a/b Test Analysis Using Python
summary
Automates assignment and analysis for dozens of concurrent A/B tests with millions of rows, producing a web app that generates code so engineers copy paste consistent toggles and prime-number interstates. Uses SQL and Redshift for extraction, runs 100,000 simulations from Beta posteriors to compute a 99.8% credible range of treatment minus control, outputs an install/maybe/avoid recommendation, and enforces censoring, exclusions, and repeat-user consistency so the system actually produces actionable decisions.
AI-written summary · about
chapters (11)
- 0:00Start
- 1:03Demystifying Statistics
- 1:24What Ab Testing Is and Why
- 2:38Causal Statements
- 5:46What Do You Need To Do in an Ab Test
- 6:25The Risk of Your Experiment
- 14:13Death Event
- 15:05Censoring
- 15:40Exclusions
- 27:33The Analysis Message
- 28:39Resources
related talks
01h 18mA/B Testing with SciPy
26mAnti-Patterns in A/B Testing - or How does AB Testing Scale?
35mThe Realities of A/B Testing
33mABBY: A Django app to document your A/B tests
33mAnalytics Data Driven Testing
59mTesting in Python
42mAutomation Testing in a Real Big Data Project
34mMaximizing marketplace experimentation: switchback design for small samples