ML Product Experiments at Scale
summary
Yelp runs a homegrown experimentation platform called Bunsen that serves ML and product tests across production, running about 700 experiments concurrently and allowing quick rollbacks. It uses MLflow for model tracking, MLeap for serialization, a Rust core allocation library wrapped per platform, and Redshift/Spark for data and inference. Outcome is operational: deterministic cohort assignment, a Beaker scorecard for lifts and guardrails, and experiments that evolve with contextual bandits.
AI-written summary · about
chapters (12)
- 0:00Introduction and speaker welcome
- 1:04Why ML experiments matter
- 3:52Pre-production ML development
- 8:46Testing vs experimentation
- 10:30Experiment design at scale
- 13:40Production testing importance
- 14:43Bunsen experiments platform
- 17:55Scorecard analysis example
- 18:35Sequential testing limitations
- 19:37Multi-arm bandit techniques
- 21:01Contextual bandits approach
- 22:25Q&A and platform details
related talks
23mLive Q&A
27mML Platform Journey
30mOperationalizing Machine Learning
44mML in Production
42mA/B testing with real-world data to evaluate ML model updates
38mFrom Experimentation to Products: The Production Machine Learning Journey
26mManaging ML Models @ Scale
32mAutomating Production Production Level Machine Learning on AWS