Experimentation Programs at Scale: Lessons Learned from Top Companies
Talkat OpenFeature Summit NA 2024 · on Tuesday, 2024-11-12· at 11:15· Salt Palace | Level 2 | 250 D·
summary
Companies report ranges from 3–5 tests per year to 50–60,000 simultaneous experiments and find only about a 30% success rate (industry averages 10–15%), so the play is volume: run many small 2–5% impact tests, instrument with feature flags to push assignments, and pre-aggregate metrics (DBT) to cut data costs that can reach millions per month. Program structures vary—central team, decentralized teams, or a Center of Excellence—and the practical goal is to bring feature flags reduce incremental cost per experiment toward zero so iteration becomes cheap.
AI-written summary · about
chapters (12)
- 0:00Experimentation programs intro
- 1:46Why companies run AB tests
- 4:36A/B test success rates
- 6:47Testing vs guessing
- 8:31Why thousands of tests
- 9:57Challenges at scale
- 11:45Cost and platform sophistication
- 13:30Crawl-walk-run phases
- 14:54A/B testing in agile
- 17:43Experiment structures at scale
- 20:31Key takeaways and advantage
- 21:18Q&A reducing costs
related talks
15mA/B Testing at Scale
36mExperimentation at Microsoft
19mThink Prod Up Front: Shifting Left on Production Excellence
53mEarly Performance Testing: Finding News You Can Use
44mLizard Optimization
50mCreating Performance Impact in a Changing Tech Landscape
25mGame time: A playbook to (unsuccessfully) 10x in a week and (successfully) 10x in a year
40mPerformance Testing and Big Data