Patterns of ML Models in Production
summary
Provides concrete patterns — pipeline, ensemble, business logic and online learning — backed by 100+ user interviews, names the real pain points (scaling one model vs many, microservice overhead, the tensor-intensity-out mismatch between I/O and compute) and shows an actual system built on Ray: Ray Serve as a scalable model‑serving framework that composes deployments, exposes handles for cheap inter‑service calls, and scales replicas across nodes so model composition and online updates run efficiently.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:41Putting ML Models in Production
- 2:45Architectural Patterns Batch
- 4:29A Typical Computer Vision Pipeline
- 7:33Pipeline Implementation
- 8:57Ensemble Use Cases
- 10:38Ensemble Deployment
- 11:54Business Logic in Action
- 13:01Key Question: Where to Run the Business Logic?
- 14:44Online Learning
- 18:56Ray Serve: Deployment
- 21:26Ray Serve: Handle
- 22:34Ray Serve: Patterns
- 22:50Business Logic in Ray Serve
- 23:54Ray Serve: A Framework for 1+ Models in Production
related talks
25mRay Serve: Patterns of ML Models in Production
41mInfrastructure for ML Applications
47mMachine Learning in Production: An Intro to MLOps.
31mMLOps In Practice – How To Run Your Machine Learning Models In Production At Enterprise Scale
44mPanel: ML Platform on Ray
25mContinuous ML Applications in Production
29mScaling ML Pipelines for Autonomous Driving with Ray
29mDemystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform