Being Intentional: AB Testing and Data Minimization
summary
Claims that standard AB-test inference (t‑tests, ANOVA, nested partial F‑tests) can run without per‑user records by storing only counts, sums, sums of squares per cell in an equivalence‑class pivot table. Describes a 2015 re‑architecture that keeps random assignment state client‑side, funnels minimal categorical metadata through an optional privacy filter, and updates fixed‑size accumulators so regressions and SRM/interference checks run cheaply at scale. Tradeoffs: no formal DP guarantees, limited use for heteroscedastic robust SEs, but large storage and compute savings for typical experiments.
AI-written summary · about
chapters (8)
- 0:00Welcome
- 1:27AB testing basics
- 3:56Random selection and stats
- 6:24Data minimization
- 8:48Aggregating data
- 11:59Benefits and trade-offs
- 13:25Monitoring with less data
- 17:17Intentional experimentation
related talks
41mA/B Testing with Style
41mLittle-Known Pitfalls of A/B Testing
32mExperimentation: putting research papers into prod
36mA/B Testing in Production
52mAB-tests, lies, damned lies, and statistics.
36mA/B Testing in Production
46mAB Test Bloody Everything
18mBuilding an AB testing analytics system with R and Shiny