What I learned from monitoring more than 30 Machine Learning Use Cases
summary
Runbook-style takeaways: monitoring comes from running 30+ production models and boils down to concrete measures not vague topic talk. Use existing stacks (Grafana, Prometheus, CloudWatch) and log predictions/features to S3. Prioritize output-first signals: live holdout evaluation (a few percent of traffic) with alert if precision drops below 95%, and latency alert like p95 latency < 300 ms. Add response-distribution checks via histograms / D1 histogram comparison, then layer input/feature drift once alerts are stable.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:17Software Monitoring Basics
- 2:43Use the four golden signals
- 6:39Silent falures causes huge commercial impact
- 10:25Practical Example: Process Cost Optimization for Private Loans
- 14:41Monitoring Priority 1: Evaluation Metrics in Production
- 16:46Monitoring Priority 1: Stakeholder Fear Signals
- 19:54Response distribution
- 23:07Monitoring Priority 2: Heuristic Quality Metrics
- 25:12Input and Feature Data distribution
- 28:32Monitoring: Example Implementation
related talks
25mSymptom-based Alerting for Machine Learning - What I Learned...
05mLightning Talk: Machine Learning Observability with Prometheus
39mAn SRE Approach to Monitoring ML in Production
42mMachine Learning In Observability Systems: From Insights To Action
18mReliable Maintenance of Machine Learning Models
26mMonitoring models in production
24mObservability in the MLOps Lifecycle with Prometheus
23mMastering Monitoring & Alerting: Key to Seamless Performance