Build Your Own ML Data Feedback Loop
summary
Describes a production loop where model suggestions are shown to doctors whose yes/no feedback becomes ground truth for retraining, with PHI scrubbed and joins from Mixpanel and StackDriver logs into BigQuery/Cloud Storage for postprocessing. Uses feature flags (LaunchDarkly) for online A/B, measures top‑K accuracy and user satisfaction, compares offline metrics to the semi‑grand 70–80 case test set, and retrains models with PyTorch/TensorFlow/pandas. Mentions infra: Docker, Kubernetes, Helm and the ethical need to store logs HIPAA‑compliantly.
AI-written summary · about
chapters (11)
- 0:00Introduction to talk
- 1:47Why use ML feedback loop
- 5:37Doctor feedback in production
- 7:44Building the feedback loop
- 10:12Data sinks and logging
- 12:39Data processing and retraining
- 14:47Testing models offline
- 16:10Testing models online
- 17:56A/B testing with feature flags
- 20:52Ethics and compliance in loops
- 22:39Q&A session
related talks
30mHow to Properly Test ML Models & Data
36mApplied Machine Learning Playbook: From Idea to Production
40mDesigning Data Driven Projects: Beyond ML Experimentation
13mHuman-in-the-Loop MLOps: Boost Model Performance by 40%
26mCompleting the machine learning loop
38mBuilding robust machine learning systems
18mReliable Maintenance of Machine Learning Models
35mA simple way to validate and monitor the performance of your ML Applications.