Deploying ML models and all the things that go wrong
summary
Deployment experience since 2017 shows concrete pitfalls: a brittle custom prediction service ends up hosting 20+ models on an unscalable architecture, causing reproducibility and retraining pain; integration gaps with backend teams break data contracts when categories change; lack of tests lets a new employment category crash preprocessing; absent monitoring hides dataset shift until a scoring distribution skews left and mass approvals occur, costing customers. Practical tools discussed include Flask/Django, Docker, MLflow and Airflow and the need to plan for scaling, delegation and continual model monitoring.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 2:06Agenda
- 2:51ML training process
- 5:59Configuration of deployment
- 7:59Deployment as a separate entity
- 10:01Predictor service
- 10:52Deployment process
- 12:03reproduce results
- 13:13testing integration
- 16:10testing
- 18:36Plan deployments thoroughly
- 19:28Why scale
- 20:17Delegation
- 23:47Failure story
- 24:58Cooperation
- 25:50Data Science
- 26:28Tests
- 28:52Lessons learned
- 30:06Not monitoring models
- 32:24Lessons
- 34:12Other mistakes
- 34:41The essence of a data scientist
- 36:35Other teams
- 37:10Data scientists
- 38:18QA
related talks
39mApplying deployment oriented mindset for building Machine Learning models
29mDeploying ML Models and the Things That Go Wrong
28mUnravelling Hidden Technical Debt in ML: A Pythonic Approach to Robust Systems
46mProduction Model Deployment
43mMachine Learning Engineering principles with Python and MLFlow
17mContinuous deployment for ML: The new software development lifecycle
59mWhy Great Models Fail: Lessons From 9 Years of Deploying ML Models
29mDemystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform