Testing Machine Learning Models
summary
Claims concrete outcomes: a model shows 95% training accuracy and 76% validation accuracy (overfitting), behavioral testing produces twice as many tests and finds ~3× more bugs, and Zillow’s automated buying system mispriced homes and lost staff when COVID shifts break assumptions. Tests demonstrated adversarial attacks (turtle→rifle, stop sign stickers), pipeline points like kinesis/s3/snowflake matter for fault tolerance, and runtime checks use locust.io performance testing plus API checks with requests and transformers-based experiments.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:40Head of Engineering
- 7:05Define the Problem
- 8:51Define Success and Assess Risk
- 10:24Design the initial Architecture
- 10:504. Collect Data
- 13:02Prepare Data
- 13:44Train and validate Models
- 16:087. Test the Models
- 20:41Lessons Learned
- 26:22What does it look like?
related talks
32mTesting Machine Learning Systems: What it is and why you should care
30mHow to Properly Test ML Models & Data
53mUnleashing the Black Box: Testing Machine Learning Systems
46mSW Testing: Can ML save us?
24mTesting Techniques for Critical Deep Learning Applications
26mTesting a Data Science Model
25mWhy Good Models Fail After Deployment
16mTransforming Test Automation with AI and Machine Learning