Monitoring machine learning models in production
summary
Shows concrete pipelines using Alibi Detect and Seldon Core plus Knative eventing to run outlier, adversarial and drift detectors on deployed models. Demonstrates a VAE outlier detector with inferred thresholds, an adversarial-correction method that restores accuracy from ~93%→~0%→82% under Carlini/Wagner and JSMA attacks on CIFAR-10, and KS-based drift tests (untrained autoencoder or softmax features) that flag corruptions that drop accuracy to ~22%. All examples run in the provided notebook and on Kubernetes.
AI-written summary · about
chapters (7)
- 0:00ML model monitoring intro
- 2:54Why monitoring matters
- 6:49Model performance metrics
- 11:42Outlier detection deployment
- 14:33Adversarial examples detection
- 23:06Data drift detection
- 30:24Summary
related talks
25mMonitoring AI with AI
26mMonitoring models in production
36mThe ML Monitoring Flow for Models Deployed to Production
30mDetect machine learning (ML) model drift in production
34mTraining & Monitoring AI - Drift Detection
40mMLSecOps with Automated Online and Offline ML Model Evaluations on Kubernetes
33mAt the age of AI and ML
18mBeyond Inference: Bringing ML into Production