AI Reliability in Security: A New Data Science Paradigm
summary
Defines mean time to decay (MTTD) as a deliverable metric and gives equations to compute an instantaneous failure rate from monitored feature drifts, converting conditional probabilities into rates, integrating to an average decay rate (ADR) and a cumulative decay rate. Recommends concrete monitors — skewness, kurtosis, variance, volumes, ROC over time, LIME, Grad‑CAM, SHAP, adversarial poisoning checks — and reports business impacts like ML Ops profit lift of 3–15% and an ML Ops market >$4B by 2025.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 0:55Three words to describe AI
- 2:55AI reliability
- 4:14Why AI reliability
- 5:18AI reliability mathematics
- 6:44Software reliability
- 9:32Questions about reliability
- 10:55Implementing monitors
- 12:15Review
- 12:59Monitors
- 23:26Summary
related talks
53mMetrics that Matter.Confidence & Resilience in Your Security Program
12mModel Monitoring: Detecting and Analyzing Data Issues
40mYour Metrics Suck! 5 SecOps Metrics That Are Better Than MTTR
21mAI Hardware Reliability at Scale
23mHow to detect silent failures in ML models
42mThe Fault in Our Metrics: Rethinking How We Measure Detection & Response
18mReliable Maintenance of Machine Learning Models
36mThe ML Monitoring Flow for Models Deployed to Production