Experimentation in Data Science
summary
Argues that experimentation is the systematic way to learn in production, not just A/B tests, and shows concrete practices: use multi-armed bandits for continuous explore–exploit, condition on key features and apply dimensionality reduction to avoid confounding, estimate sample size and monitor stopping rules to save users, and drive tooling to lower cost per question. Notes industry examples from Google’s thousands of experiments to clinical trials and calls out expensive enterprise tools like Optimizely as blockers.
AI-written summary · about
chapters (16)
- 0:00Start
- 0:24Experimentation in Data Science
- 6:18Why Should Humans Care about Experimentation
- 12:34William Gossett
- 18:25Industries Where Experimentation Is Incredibly Important and Useful
- 27:27Explore Exploit Framework
- 29:41Cost of Running an Experiment
- 33:43A / B Testing
- 33:56A / B Test
- 34:27Randomized Assignment
- 35:23Confounding
- 41:45Experimentation Design Process
- 46:03Experiment Design Process
- 47:07Ignoring Data
- 47:53Dimensionality Reduction
- 50:44Clean Experiment Design Process
related talks
24mThe Scientific Method of Experimentation
18mHow Data Scientists Broke A/B Testing (And How We Can Fix It)
13mProduct Experimentation for Growth
43mAgile Strategies for High-Impact Data Science & Machine Learning Projects
41mKeynote: Personalization, Bandits, and Causal inference
01mintro video
25mExploring Tradeoffs in Experiment Management as part of AI Platforms
50mData-Driven Engineering