Reliable Machine Learning
summary
Machine learning introduces new failure modes across data pipelines, models and serving code, and practical fixes are concrete: assert upstream data with create expectations to catch the Excel gene-to-date bug that affected ~20% of cases, enforce feature types and schemas (Salesforce-style) to avoid misinterpreting fields, use metamorphic testing to check clustering without labels, monitor input/output drift and calibration, apply LIME for per-prediction explanations, and expose small user controls so post-processing and knobs reduce user frustration and operational risk.
AI-written summary · about
chapters (21)
- 0:00Intro
- 0:26Pre-roll: leading machine data analytics service
- 3:57Yet another Google resource
- 4:51Types, tests, and enumerating implementations
- 12:19Testing your data
- 12:48Example test assertion
- 17:17Calibrated predictions
- 18:01Actual model training and serving
- 18:51Types in general
- 20:19Information flow control with types (SBTB 2018)
- 21:39Challenges of testing ML
- 22:42Property testing: practice
- 25:59Metamorphic testing: key references
- 26:25Monitoring Input/Output
- 26:43Debugging / troubleshooting
- 27:39Human psychology and algorithmic systems
- 28:20Example: Sumo Logic LogReduce feature
- 29:19Problem: pen doesn't work in space / zero-G
- 29:37ML as a hammer looking for nails
- 30:03Inferring latent multiple users from Netflix data
- 30:55Three key take aways
related talks
18mReliable Maintenance of Machine Learning Models
21mHow The Application Of Machine Learning Is Changing The Way Companies Use Data
31mReliable Data for Large ML Models: Principles and Practices
28mHindsight bias: A machine learning story
48mPrinciples and Practices of Building Machine Learning Systems
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
44mSRE and ML: Why It Matters
29mDemystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform