Best Practices for Prototyping ML Models for Healthcare
summary
Practical checklist for prototyping ML on EHR: define cohorts by prediction time and discard future data to avoid temporal leakage, split train/test by time and by patient, and audit labels (30‑day mortality often <1% while readmission ≈15%). Clean data by distinguishing anomalies from outliers — albumin example: mean 3.5 g/dL sd 0.65, negative values are anomalies — and prefer precision‑recall over ROC for very imbalanced tasks — ROC misleading on <1% mortality, use precision‑recall — compare against benchmarks since deep learning only 1–2% better than logistic regression on structured EHR.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 0:58Data Science and Machine Learning
- 2:18Motivations for Machine Learning
- 4:56Electronic Health Records
- 5:49Time
- 6:17Temporary Solution
- 8:11Binary classification
- 8:47Core definition
- 10:08Data Quality
- 10:20Lab Results
- 11:05Date of Death
- 12:13Anomalies vs Outliers
- 14:40Innate Training Testing Split
- 16:02Performance Metrics
- 16:24Data Background
- 17:04Performance Matrix
- 18:21Misconceptions
- 19:35Precision Recall Plot
- 19:49Mortality Prediction Example
- 21:35Good Guidelines
- 22:38Benchmarks
- 24:24Comparison with benchmark
- 25:10Survival analysis
- 26:23Time to events
- 27:59Survivor curves
- 28:24Comparing survival curves
- 29:30Final remarks
- 30:30Questions
related talks
36mApplied Machine Learning Playbook: From Idea to Production
27mBuild Your Own ML Data Feedback Loop
35mML: Beyond confusion matrix, model performance analysis
35mResponsible implementation of Machine Learning models for life-and-death decision making
31mReliable Data for Large ML Models: Principles and Practices
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
11mDesigning Governed AI Pipelines for Healthcare Impact
18mMedical AI Beyond Prediction: From Normative Learning to Agentic Healthcare AI