Introducing Ray Serve: Scalable and Programmable ML Serving Framework
summary
Ray Serve is a distributed Python web framework on Ray that replaces Flask-or-TFServing tradeoffs by letting HTTP APIs own preprocessing/postprocessing while offloading inference to backend with replicas (configurable CPUs/GPUs and batching). It targets slow models that do ~10–100 QPS by enabling transparent scaling, batching (e.g., small batches of 4 give big throughput gains), and composition via python-reachable handles so endpoints call other endpoints and run ensembles. It supports 70/30 traffic split routing for canary/A-B, runs locally, on Kubernetes or cloud, and is pip-installable.
AI-written summary · about
chapters (7)
- 0:00Introduction to ML serving
- 1:21Requirements for model serving
- 4:20Traditional serving methods
- 8:36External serving challenges
- 13:00Introducing Ray Serve
- 17:16Advanced model composition
- 21:44Deployment and conclusion
related talks
25mRay Serve: Patterns of ML Models in Production
21mAchieving Scalability and Interactivity with Ray Serve
26mRay: A System for High-performance, Distributed Python Applications
31mFaster Model Serving with Ray and Anyscale
24mRay: A System for High-performance, Distributed ML Applications
22mRay, a Unified Distributed Framework for the Modern AI Stack
15mRay: A System for Distributed Applications
32mAn Overview of CloudKitchens's Ray-Powered ML Platform