ML in Production
summary
A concrete account of shipping ML: a COVID forecasting pipeline for HHS that predicts multiweek peaks for 3,000 locations by running daily full retrains and completing end-to-end runs in about eight hours. It ingests ~30 datasets expanded into ~50 aggregates, stores backups on S3, uses SQLAlchemy tables for dataset versioning, automated Jenkins jobs, backtesting, sanity checks and manual approvals, and required many data checks and ad hoc fixes to remain reliable while handling about 100 TB/day.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 3:57MLOps vs DevOps
- 13:51Model monitoring
- 20:15ML maintenance example
- 27:10Data versioning & backups
- 32:23Daily retraining benefits
- 33:27Q&A
related talks
29mML in Production on Databricks
25mML in Production–What does “Production” even mean?
34mML and analytics addressing nationwide COVID-19 impact and recovery
29mDemystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform
25mContinuous ML Applications in Production
27mML Platform Journey
10mML-based Log Analysis for Faster Debugging
21mML Infrastructure for Autonomous Vehicles