Unleashing the Black Box: Testing Machine Learning Systems
summary
Claims that 87% of ML projects never reach production and that TDD can enforce FATE drive a checklist of concrete techniques: use SMOTE balancing to equalize classes (demo shows class counts equalized), run unit/integration/model/deployment tests with pytest, measure model metrics (a demo sentiment classifier reports ~80% accuracy on small data), compute equal opportunity difference = 0 for a loan model, and generate LIME explanations for feature-level rationale; practical mitigations for technical debt, monitoring and retraining pipelines are shown and tested.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:30Announcements
- 4:50Bio
- 5:05Agenda
- 5:47What is Machine Learning
- 10:03Core Problem
- 13:28Root Problem
- 15:40Strategies
- 19:20ML Test Pyramid
- 26:52ML Fairness
- 31:16Who should be responsible
- 33:18Trust is critical
- 34:47Strategies to build trustworthy systems
- 37:21What is explainability
- 38:30Demo
- 40:07Technical Debts
- 42:59Solutions
- 46:11Conclusion
- 47:35QA
related talks
39mBeyond Bias: Crafting Fairness in Pythonic Machine Learning Systems
42mMasterclass: The right path to ML
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
28mLeading machine learning research journeys with happy (into-production) endings
32mTesting Machine Learning Systems: What it is and why you should care
28mML-Driven Test Intelligence at Scale - What Works What Fails and Why It Matters
27mPyTesting the Limits of Machine Learning
31mTesting Machine Learning Models