SRE for ML: The First 10 Years and the Next 10
summary
Built a production ML stack for ad selection since ~2009: data ingestion and preprocessing, training cluster, a model management and pipeline orchestration layer, a quality-checking system, and model serving. Key operational claims: many surprising outages and quality drops trace to infra (data distribution shifts, pipeline stalls, resource exhaustion, model-load failures) not model math; promotion-to-production uses concrete promotion-to-production checks (training reliability, data slices, resource/use metrics) and automated retrain-history gating. Roadmap: converge platforms, drop costs, make model quality evaluation standard so ML reliably enables fast iteration.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:23Agenda
- 4:17Basic Questions (the usual)
- 7:44ML vs. AI: No, Seriously
- 8:57What even is ML?
- 10:01ML: Learn from Examples (simple supervised example)
- 12:36The Beginning: SRE for ML
- 16:04(our) Basic ML Systems Architecture
- 21:31New Teams, New Models
- 23:13The ML Platform
- 25:49One Elusive Goal
- 32:57Big Idea
- 35:09What Metrics Mean It Works ?
- 37:19Possible(?) Automation
- 38:33Speed Matters
- 39:53Make ML Boring
- 42:31SRE Teams Will All Do ML
related talks
44mSRE and ML: Why It Matters
39mAn SRE Approach to Monitoring ML in Production
42mImproving Machine Learning Development Reliability
40mProductionizing Machine-Learning Services: Lessons from Google SRE
32mTeam Teardown: Evolving an ML Platform
47mMachine Learning in Production: An Intro to MLOps.
25mBuilding Scalable MLOps Pipelines That Actually Work
41mInfrastructure for ML Applications