AB-testing by cluster
summary
Concrete claims and code: Python notebooks that run t-test based A/B and A/A batch simulations, compute p-values and confidence intervals, and measure minimum detectable effects by sensitivity testing. Examples: 1,000 observations detects ~7.8% change, 10,000 ~0.5%, 100,000 ~0.28%. Clustering by city is implemented and shows how large imbalanced clusters destroy false‑positive calibration; simulating many random splits gives a practical power estimate and required uplift (often 20–50% for small samples). Repo exists and the simulations run, exposing when tests are unusable.
AI-written summary · about
chapters (9)
- 0:00A/B testing basics
- 5:56Defining A/B tests and metrics
- 13:12Statistical tests assumptions
- 21:17A-tests for false positives
- 27:11Sensitivity testing
- 39:24Clustered A/B testing pitfalls
- 51:26Balancing tests segmentation
- 1:06:27Communicating results
- 1:14:47Q&A
related talks
37mA/B testing at Zalando: concepts and tools
44mA B Testing: Harder than just a color change
52mAB-tests, lies, damned lies, and statistics.
39mExperimentation and the gold standard of data champions
29mIt might look normal but this distribution will ruin your stats
24mEffective Product Experimentation: Using Data-Driven Techniques for Iteration
24mBeyond the P-Value: A Data Scientist’s Guide to Tier-1 Feature Launches
20mHow to Win Friends and Influence People (With Data)