How to Test a learning system
summary
Shows a concrete workflow: Vision AI (a single‑shot YOLO variant with a patent) is validated against ~800,000 labeled entries and an 85% accuracy requirement, with discovered failure modes like 99% overall but 70% for underrepresented groups. Uses combinatorial testing (Tosca) to collapse 24,590 meta‑feature intersections to 149 critical cases, flags any failing cell to block production, and traces errors to dataset bias, feature categories, or low‑resolution/tree UI patterns.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:35Presentation
- 6:13What are learning systems
- 6:53What is this
- 8:18How did you know
- 9:42Example
- 10:22Neural Networks
- 11:21Backward Propagation
- 12:19The Problem of Testing
- 14:46Assess
- 15:45Testing Example
- 16:13Testing Advice
- 16:54GDPR
- 18:07Risk
- 19:19Testing
- 19:37Requirements
- 20:06Accuracy
- 24:39Inaccuracy
- 28:04Case Study
- 35:47conclusion
related talks
01h 08mMastering Visual AI with Vision-Language Models & Advanced Evaluation Techniques
24mHow to Test AI Features Before Your Users Do
46mSW Testing: Can ML save us?
31mTesting Machine Learning Models
34mTesting Live Video Streams At Scale
01h 01mWe are the Guardians of our Future
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
33mRunning Machine Learning in Production - a Journey to Success