Why Good Models Fail After Deployment
summary
Claims that 87% of data science projects never reach production and Zillow loses ~\$1B when pricing models fail; real failures include Google’s diabetic retinopathy model dropping under low‑quality clinic images and COMPAS amplifying bias. Detection uses PSI, KS, Jensen-Shannon for drift and class‑distribution tests for label shift. Remedies implemented include feature store, robust data validation (Great Expectations), automated monitoring and retraining strategies (time, performance, drift, hybrid), and observability platforms like Evidently AI, Fiddler, Arize.
AI-written summary · about
chapters (9)
- 0:00Why Great ML Models Fail in Production (Intro + Agenda)
- 1:24Failure Mode #1: Data Drift — What It Is, Examples, Detection & Fixes
- 4:14Failure Mode #2: Concept Drift — When the World Changes the Rules
- 7:10Failure Mode #3: Label Drift — Shifting Base Rates & Recalibration
- 10:45Failure Mode #4: Feature Pipeline Degradation & Training-Serving Skew
- 15:20Failure Mode #5: Feedback Loops & Bias Amplification
- 19:17Building an ML Monitoring Dashboard: Performance, Data Health, System Health
- 21:13Retraining Strategies: Time-Based vs Performance vs Drift vs Hybrid
- 23:02Production ML Best Practices + Final Takeaways & Outro
related talks
59mWhy Great Models Fail: Lessons From 9 Years of Deploying ML Models
30mWhen Good Models Go Bad: The damage caused by wayward models and how to prevent it
34mData Science beasts (failures) and where to find them
56mData, AI, Hype and Fear - Recipe for Disaster?
44mKeep it Clean: Why Bad Data Ruins Projects and How to Fix it
08mConsiderations for producing the best AI and Machine Learning models
19mModel Performance Monitoring - A Practitioner's Perspective
35mWhat Failure Taught Me About Building High-Stakes Models