From 0 to 10,000 experiments
summary
Claims a path from 0 to 10,000 experiments by combining feature flags, an experimentation platform that standardizes statistical evaluation, and automated diagnostics that catch biased assignment at launch. Implements global holdout to reconcile five separate +5% lifts that don’t add up, and enforces mutual exclusion so conflicting promos never reach the same user. Automates canary deployments with auto-rollbacks across users, pods, geos, and uses contextual bandits for audience-specific wins, reporting thousands of commits a week and a measurable uplift.
AI-written summary · about
chapters (11)
- 0:00Correlation not causation
- 1:07Randomized control trials matter
- 2:11Simple A/B tests
- 3:14Culture of experimentation
- 3:55Experiment reliability
- 5:21Decision criteria for tests
- 6:22Beyond basic A/B testing
- 7:48Advanced experimentation techniques
- 8:30Canary deployments
- 9:5310,000 experiments
- 10:34Making experimentation accessible
related talks
39mExperimentation and the gold standard of data champions
27mDEVOPS DAYS MTL 2018 - Lenore Adam
16m⚡ Lightning Talk: The Permutations and Combinations of Managing Features: Beyond the Toggle
16mAdvanced Feature Flagging: It's All About The Data
28mExperiments for SRE
53mDeliver Results, Not Just Releases: Control & Observability in CD
24mEffective Product Experimentation: Using Data-Driven Techniques for Iteration
24mScaling on Satisfaction: Automated Rollouts Driven by User Feedback - Thomas Vitale & Whitney Lee