Learning from pilots
summary
Explains how small pilots (often ~100–200 merchants) give weak signals but can be strengthened using bootstrapping to show sensitivity, building digital twins with propensity scores to reweight hundreds or thousands of customers, or applying multi-level regression and poststratification (MRP) to borrow strength across hierarchy. Quantifies limits: opt‑in bias, unmeasured confounders, rare events and engineering noise, and recommends randomize when possible or negotiate larger pilots and stakeholder alignment.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:32Ideal A/B testing world
- 5:55Post-pilot analysis challenges
- 9:46Statistical magic for pilots
- 17:48Hierarchical pilots insights
- 23:39Workflows for pilot insights
- 27:52Non-randomized alternatives
- 31:00Key takeaways
- 32:25Q&A
related talks
19mScaling AI: From Experimentation to Transformation
32mScaling a Responsible Agentic Workforce
35mWhat Failure Taught Me About Building High-Stakes Models
18mData Science Careers via A/B Testing
19mVoices from the Skies: Real world lessons to help drive Copilot adoption and ROI
22mSetting Up Your Growth Team To Scale
24mConditions for continuous transformation
28m"The Top 2%: How to Identify and Build for Your Most Powerful Users"