Ray Serve: Patterns of ML Models in Production
summary
Ray Serve is a Ray-based serving framework developed from 100+ user interviews that claims to deploy many models as composable microservices with per-component scaling, GPU and batching support, and gRPC actor-to-actor calls to avoid heavy serialization. It exposes a Python API with serve handle for nested calls that preserve Python types, separates IO-heavy business logic from compute-bound inference, and offers FastAPI integration so existing HTTP routes and type validation scale across replicas.
AI-written summary · about
chapters (16)
- 0:00Introduction to Ray Serve
- 1:22What is Ray Serve
- 2:46Ray Serve in Ray ecosystem
- 3:30ML serving challenges
- 5:58Four ML deployment patterns
- 6:19Pipeline pattern explained
- 9:52Implementing pipelines in Ray Serve
- 13:20Ensemble pattern overview
- 15:48Implementing ensembles in Ray Serve
- 16:08Business logic in ML serving
- 19:19Separating IO and compute
- 20:21FastAPI integration
- 21:26Online learning pattern
- 22:50Ray Serve core concepts
- 23:34Production use cases
- 24:37Conclusion and resources
related talks
23mIntroducing Ray Serve: Scalable and Programmable ML Serving Framework
26mPatterns of ML Models in Production
31mFaster Model Serving with Ray and Anyscale
22mRay, a Unified Distributed Framework for the Modern AI Stack
31mKlaviyo's Journey to Robust Model Serving with Ray Serve
26mRay: A System for High-performance, Distributed Python Applications
24mRay: A System for High-performance, Distributed ML Applications
15mRay: A System for Distributed Applications