Monitoring AI with AI
summary
Model deployments include an immutable package that bundles training data statistics (min/max, quantiles, autoencoder snapshots) and run-time checks like Kolmogorov–Smirnov, PCA and density clustering. Anomalies are detected by parallel models—deep autoencoders, GAN discriminators and histogram comparisons—then surfaced as alerts; a proof‑of‑concept on a Berkeley single‑shot detector flags previously unseen traffic‑light appearances in production. Integration uses service mesh / Envoy for traffic shadowing and a feedback loop that feeds flagged cases into retraining.
AI-written summary · about
chapters (13)
- 0:00Start
- 1:36ML Model in production
- 2:19Model Observability and Reliability
- 3:31Original modeled concept
- 3:56Uncertainty - a blind blank space
- 4:22Gradient edge cases
- 4:34New Concept
- 5:45Data "exploration" in production
- 12:58Add K-S test, Q-Q plot and PCA on top
- 14:30Classical Autoencoders for anomaly detection
- 15:57GANs for concept drift detection
- 17:00Variational Autoencoders (VAE and CVAE)
- 17:27Masked Autoencoder for Density Estimation (MADE)
related talks
31mMonitoring machine learning models in production
29mAI-Powered Anomaly Detection in Cloud Data Pipelines
19mThe Rise of AI in Anomaly Detection
08mBroadening AI Adoption with AutoML
34mTraining & Monitoring AI - Drift Detection
26mMonitoring models in production
25mMonitoring AI Pipelines Output As Product
42mAI Math Models Help Hunting Cybersecurity Threads