How to Properly Test ML Models & Data
summary
Claims continuous validation across training to deployment and a working open source stack that automates checks for data, splits and model behavior. Concrete examples include 112000 x‑rays from 30000 patients with train/test patient overlap, a bot detector tuned to recall >0.99 precision ~0.8, a retrain‑daily recipe using the last eight days, and a production bug where an API changed calibration from 0–1 to 0–10. Demonstrated tooling: Deepchecks and Great Expectations plus pytest, hypothesis, Airflow, GitHub Actions and CircleCI.
AI-written summary · about
chapters (8)
- 0:00ML model testing importance
- 1:53ML vs software testing
- 5:01Testing ML pipelines
- 14:55ML model problems
- 20:26Systematic ML testing
- 24:38When to test ML models
- 26:46ML testing tools
- 28:13Summary on ML testing
related talks
35mA simple way to validate and monitor the performance of your ML Applications.
18mReliable Maintenance of Machine Learning Models
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
31mTesting Machine Learning Models
39mAn SRE Approach to Monitoring ML in Production
27mBuild Your Own ML Data Feedback Loop
26mTesting a Data Science Model
53mUnleashing the Black Box: Testing Machine Learning Systems