Monitoring AI Pipelines Output As Product
summary
Augury monitors ML pipeline outputs as product by instrumenting the detection lifecycle end-to-end: ~50 million machine-hours, ~80,000 machines, dozens of detectors feed a detection management layer that filters 99.87% of alerts. The team adds metrics in Graphite, dashboards and Grafana alerts, tracks per-detector counts (started with a bearing-wear detector), posts to a Slack channel, and stamps graphs with deployment tags to pinpoint code or ops changes; this setup catches bad deployments and security changes before customers see them.
AI-written summary · about
chapters (9)
- 0:00Monitoring AI pipelines
- 1:41Augury’s mission
- 3:50Machine health AI data
- 8:23AI detections in products
- 10:28Detection management role
- 14:59Goals and challenges
- 18:12Hybrid monitoring approach
- 20:40Proactive Grafana alerts
- 24:56Conclusion and key takeaways
related talks
25mMonitoring AI with AI
29mAI-Powered Anomaly Detection in Cloud Data Pipelines
19mThe Rise of AI in Anomaly Detection
40mOptimizing AI Applications for Production with Observability
12mUnleashing the Power of AI for Performance Monitoring& Anomaly Detection
01h 21mBuilding a ML Model Observability pipeline
40mKeep AI on Track: Guardrails + OpenTelemetry Observability
48mAIOps: New Approaches to Reduce Alert Noise