The specifics of the AB-test for Unit Economics with a network effect
summary
Randomized A/B on per-order unit economics breaks when shared fixed costs make group outcomes interdependent: a variant can look better per-order in the slice but produce worse overall profit once applied to the whole customer base. Shows a concrete numeric paradox from retail and explains why fixed costs bind groups, why switchback and clustering often don’t apply, and how to replace naive comparisons with a simple predictive model of unit_econ as a function of N using linear or log approximations for variable costs to evaluate true lift.
AI-written summary · about
chapters (9)
- 0:00AB tests and unit economics
- 2:29AB tests and network effects
- 5:19Network effects disrupt AB tests
- 7:05Network effects in social networks
- 10:39Unit economics for retailers
- 14:14Hidden network effect in AB tests
- 17:47Fixed costs paradoxes in tests
- 21:39Handling network effects in tests
- 23:27Q&A
related talks
18mHow Data Scientists Broke A/B Testing (And How We Can Fix It)
26mAnti-Patterns in A/B Testing - or How does AB Testing Scale?
27mHow to A/B test with confidence
28mThe Hitchhiker's Guide to Advanced A/B Testing Techniques
29mBalancing Personalization & A/B Testing in a Customized World
15mA/B Testing: Test Your Own Hypotheses & Prepare to be Wrong
21mLeveraging Multi-Armed Bandit Algorithms for Dynamic Decision Making
47mIncrementality Bidding and Attribution