Model Evaluation for Humans
summary
Provides concrete evaluation steps: frame success as a business metric (profit or dollars), set a deployable last-week rating baseline, check for target leakage and unexpected top features, and run cross-validation that mirrors deployment (train-on-month predict-next-month). Tries heavier models like LSTM/CNN on a cleaner national dataset before noisy production data, deploys with model IDs via REST for A/B tests, and monitors inputs for covariate shift preferring monitor ranges not KL alerts and simple sanity checks.
AI-written summary · about
chapters (12)
- 0:00Welcome
- 1:05Model evaluation in industry
- 2:08Evaluation across lifecycle
- 4:41Setting up evaluation
- 8:37Pre-deployment and iterations
- 10:42Best practices in development
- 13:11Baseline model
- 17:29Cross-validation
- 20:39Model usefulness in production
- 21:21A/B testing for comparisons
- 22:43Monitoring production models
- 25:14Conclusion and Q&A
related talks
19mModel Performance Monitoring - A Practitioner's Perspective
35mA simple way to validate and monitor the performance of your ML Applications.
30mHow to Properly Test ML Models & Data
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
29mAutomated Model Update & Evaluation
18mReliable Maintenance of Machine Learning Models
36mThe ML Monitoring Flow for Models Deployed to Production
20mFive hard earned lessons about Evals