Experimentation: putting research papers into prod
summary
Claims that interleaving yields two orders of magnitude more sensitivity than classic A/B testing and that Netflix Amazon and Airbnb use interleaved comparisons drive the practical angle: build a reusable interleaving library, add item‑level logging and client instrumentation, and measure performance impacts when invoking multiple rankers. Google’s overlapping experiments paper supports running overlapping tests to multiply test capacity by layering independent product surfaces. Also prioritize feature‑level, high‑sensitivity metrics so experiments finish in weeks instead of months.
AI-written summary · about
chapters (12)
- 0:00Experimentation and A/B testing
- 2:12Why AB testing is crucial in product development
- 5:43Why A/B testing skipped
- 9:15Rapid ML ranking iteration
- 13:32Interleaved comparisons in practice
- 17:03Insufficient user testing capacity
- 20:35Overlapping testing strategies
- 23:24Metrics not moving in tests
- 26:15Feature level metrics insights
- 28:02Solutions to A/B testing challenges
- 29:49Why innovate in experimentation
- 31:56Q&A
related talks
32mExperimentation: putting research papers into prod
28mOptimizing your A/B testing roadmap
24mEffective Product Experimentation: Using Data-Driven Techniques for Iteration
28mExperimenting your way to a better product
36mA/B Testing in Production
01h 15mExperimentation at scale: Fuel your application’s growth through A/B testing
34mA/B Testing You'll Like
52mBuilding an A/B Testing Framework that works for Developers