How to review a model
summary
Models are framed as a high‑risk technical‑debt asset and the claim is that disciplined review practices pay that debt down. The repo contains notebooks, a rubric and concrete tools: provenance tracking (W3C prov, Python prov tools), extract‑transform‑load datasets, imputation tests that break feature separations, replaying old requests to detect drift, wrk benchmarks and monitoring with Papertrail Rollbar Sentry Kafka and model servers like Bento MLflow RedisAI. An example shows 120 samples with most learning achieved by half the data.
AI-written summary · about
chapters (8)
- 0:00Model review journey
- 3:16Model deployment environments
- 8:51Data contracts and assumptions
- 13:26Model calling patterns
- 22:11Model documentation expectations
- 29:18Missing data strategies
- 32:27Debugging live models
- 35:17Summary and next steps
related talks
27mThe Model Review: improving transparency, reproducibility, & knowledge sharing...
30mWhen Good Models Go Bad: The damage caused by wayward models and how to prevent it
30mHow to Properly Test ML Models & Data
19mModel Performance Monitoring - A Practitioner's Perspective
20mBetter Care and Feeding of Machine Learning Models
18mReliable Maintenance of Machine Learning Models
01h 19mModel criticism with ArviZ
32mMachine Learning Models Versioning Using Open Source Tools