Hidden engineering challenges of A/B Testing
summary
Explains concrete engineering costs: using Optimizely/Firebase increases cyclomatic complexity (nested enroll checks, fallback or crash decisions), forces maintenance for iOS 16 vs iOS 17 variants, and produces experiment-induced damage like recreating a home view controller that discards an in-progress checkout. Requires extra infra to join crash logs with experiment assignments, expands test matrix so pass coverage explodes, and shows industry rates where big teams run 10,000 AB tests per year with only ~10–20% clear wins.
AI-written summary · about
chapters (12)
- 0:00Introduction and speaker
- 1:52What is A/B testing
- 2:56Code complexity in A/B tests
- 9:23Experiment-induced damage
- 11:09Data corruption risks
- 13:23Testing complexity of A/B tests
- 20:52Debugging A/B test issues
- 23:34Developer experience challenges
- 27:10Most experiments fail
- 29:42User experience impact
- 34:01Ethical A/B testing
- 35:25Conclusion and Q&A
related talks
36mA/B Testing in Production
28mOptimizing your A/B testing roadmap
36mA/B Testing in Production
45mHow we killed 80% of features and increased outcomes of a/b testing by 100%
28mBeyond A/B: Test automation approaches for web variants from A to Z
09mDemystifying the Dreaded A/B Test
55mKeynote: Experimenting in a Sociotechnical System in Software Engineering
33mHow to A/B test with confidence