I have no clue what my model is doing...
summary
Evaluates how to measure model accuracy in production when labels are missing partial or delayed demonstrating CLT and bootstrap sampling to get confidence bounds and a beta–binomial Bayesian updating workflow (start with beta(1,1), update daily with successes/failures, reset for drift), notes using UMAP for clustering and positive‑unlabeled techniques to detect label issues, describes e‑commerce customs classification and clinical risk deployments and reports that sometimes models cannot deliver value without better data or process changes.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker intro
- 1:06Real-world model evaluation challenges
- 3:34Central limit theorem and bootstrapping
- 7:05Bayesian inference for unknown ground truth
- 13:26Cost of errors beyond simple metrics
- 15:10Building trust in clinical data
- 17:19Summary and key takeaways
- 18:46Q&A: Delivering value with models
- 20:13Q&A: Estimating error costs with SMEs
- 22:57Q&A: Data quality and unexpected results
- 24:43Q&A: Other statistical estimation methods
related talks
40mModel Upgrade Schemes- Considerations for Updating Production Models
36mThe ML Monitoring Flow for Models Deployed to Production
01h 33mUnderstanding and diagnosing your machine-learning models
23mHow to detect silent failures in ML models
35mML: Beyond confusion matrix, model performance analysis
22mHow to Detect Silent Failures in ML Models
37mCreating correct and capable classifiers
27mModel Evaluation for Humans