Waking the Data Scientist at 2am: Detect...
summary
Explains how SageMaker Debugger captures tensors to detect vanishing gradients and “loss not decreasing” and can automatically stop a training job or send an SNS page; shows a few lines of code to add rules and check intervals. Describes Model Monitor using data capture config to sample (example: compare every 500 predictions), store JSONL in S3, run Spark processing jobs with the open-source dq library, and raise CloudWatch alarms that trigger retrain workflows.
AI-written summary · about
chapters (19)
- 0:00Intro
- 2:07Data Science on AWS – Book Outline
- 5:52Challenges with Machine Learning Training
- 8:03An example: vanishing gradients
- 8:55Introducing Amazon SageMaker Debugger Training data analysis, debugging, & alert generation
- 10:22Add Debugger to Training Job
- 13:24Introducing Amazon SageMaker Model Monitor Continuous monitoring of models in production
- 15:30Enable Data Collection for SageMaker Endpoint
- 16:07View data collected from endpoint
- 16:43Example saved prediction request & response
- 18:043. Create baseline with train /validation dataset
- 18:18Create a baseline
- 21:07Baselining results - Statistics
- 22:29Baselining results - suggested constraints
- 23:03Create a monitoring schedule
- 24:18View Monitoring Jobs
- 24:30View monitoring results
- 24:41Take corrective actions 1. Set alarms in Amazon CloudWatch and triggers for retraining
- 25:19References
related talks
28mGain better insights while training ML models with SageMaker Debugger
30mTrain ML models faster with better insights using Amazon SageMaker Debugger
37mStreamlining Machine Learning Operations
37mStreamlining Machine Learning Operations
30mDetect machine learning (ML) model drift in production
34mLat Designing and Building Complex Machine Learning....
33mAt the age of AI and ML
31mTrain and tune ML models to the highest accuracy using Amazon SageMaker