Detect machine learning (ML) model drift in production
summary
Deploys an XGBoost movie‑recommendation model trained on MovieLens, enables SageMaker data capture, generates baseline statistics.json and constraints.json, and runs a daily processing job that emits CloudWatch metrics and writes violations to S3 and Studio. The demo shows endpoint creation (~7 min) and processing (~6 min), detects a type drift — 100% violation for 'age' (float vs int), and wires CloudWatch → SNS → Lambda to automatically kick off retraining, demonstrating the loop works end to end.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:06Importance of model monitoring
- 3:24SageMaker monitoring features
- 4:49End-to-end monitoring flow
- 6:10Data quality monitoring
- 8:47Model quality monitoring
- 10:42Analyzing monitoring results
- 11:55Taking remedial actions
- 13:41Demo overview and setup
- 15:34Training data baseline
- 17:11Endpoint deployment
- 19:10Capturing inference data
- 21:57Generating baselines
- 24:55Detecting data drift
- 27:22Automated retraining
related talks
43mMachine Learning Ops: DevOps for Data Science
36mThe ML Monitoring Flow for Models Deployed to Production
18mReliable Maintenance of Machine Learning Models
31mMonitoring machine learning models in production
37mStreamlining Machine Learning Operations
16mMLOps — Understanding and Fighting Model Degradation in Production
19mDrive transformation through machine learning
27mScaling ML to the next level: Hosting thousands of models on SageMaker