When a platform isn’t enough: Building an experimentation culture
summary
Companies see only 4 of 20 tests succeed (roughly 10–33%), so the talk argues that a platform alone is insufficient: platforms cut engineer and analyst time and can apply variance reduction, but culture and process decide whether tests actually run, use the right near-term metrics, and connect to revenue. Common failure modes are setup issues like traffic imbalance / sample mismatch or AA deployments, plus noisy or lagged metrics that make results untrustworthy and learning hard to scale.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 2:12Why experimentation matters
- 5:01Healthy experimentation signals
- 8:09Blockers to experimentation
- 15:07Sharing learnings
- 16:53Trust in A/B tests
- 19:17Conclusion and key takeaways
related talks
31mBuilding a Data Experimentation Platform Without a Platform Team
23mBeyond the Buzz Building Developer Platforms That Drive Business Success
59mLessons from Running Developer Platforms & Communities
43mEngineering Platforms Evolved
26mLessons Learned from 7 years of Running Developer Platforms
01h 04mBuilding on Success, Beyond the Obvious
54mDeveloper experience platforms
36mShifting Left: How to Build a Culture of Testing