A/B testing with real-world data to evaluate ML model updates
summary
AWS Activate builds a personalization recommender that compares a legacy OpenSearch rules engine (control) against two ML-driven variants, using CloudWatch Evidently for traffic splitting and safe rollouts. Experiments run 30 days with a 50/30/20 split (control/model1/model2), plus AA and override checks and scheduled launches; KPIs stream to a custom pipeline and Evidently supports anytime p-value stopping. Results show a large uplift versus the rules baseline; the mixed ML+rules model wins in business value despite the popularity‑only variant having the highest click rate.
AI-written summary · about
chapters (12)
- 0:00Introduction and welcome
- 1:27Startup challenges and AWS Activate
- 12:40Building ML-based recommendations
- 15:54Why ML models need updates
- 17:22Pre-A/B test checks and AA testing
- 20:44Safe deployment and masking features
- 23:20A/B testing fundamentals
- 24:26Homegrown vs third-party A/B testing
- 28:21AWS CloudWatch Evidently overview
- 33:52Experiment setup and architecture
- 38:15Results and model selection
- 40:40Future initiatives and wrap-up
related talks
26mML Product Experiments at Scale
18mData Science Careers via A/B Testing
30mOperationalizing Machine Learning
35mHow implementing ML ended up with an if-else
40mNext Best Action: Maximizing Customer Value
39mGrowing user engagement with RL-driven personalisation
21mAWS On Air ft. Amazon SageMaker Inference Recommender
28mOptimizing your A/B testing roadmap