Faster Model Serving with Ray and Anyscale
summary
Ray Serve provides Python-native deployments with fractional GPUs, model multiplexing and integrations like vLLM, TensorRT LLM and NVIDIA Triton, enabling multi-GPU and multi-node LLMs and hybrid cloud bursting. Customers report 50% annual inference cost saved and a production recommendation cluster of 4,000 GPU nodes. AnyScale adds observability, incremental rollouts that avoid double-buffering GPUs (example reduces 20→11 GPUs), faster image pulls and streaming-safe-tensors so a 70B model achieves about 5x faster 70B model startup.
AI-written summary · about
chapters (7)
- 0:00Introduction and overview
- 3:14Motivation behind Ray Serve
- 8:14Ray Serve for GenAI applications
- 14:40Ray Serve on AnyScale platform
- 23:08Production readiness features
- 28:24Performance optimizations in AnyScale
- 30:10Q&A and closing
related talks
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
23mIntroducing Ray Serve: Scalable and Programmable ML Serving Framework
20mThe Quick Journey to Using Ray: How We Implement Ray and Anyscale to Speed up our ML Processes
16mFrom Self-Managed Ray to Anyscale Services: Migrating ML Serving
13mRay and Anyscale: An Optimization Journey
25mRay Serve: Patterns of ML Models in Production
28mScaling Multimodal Data and Reinforcement Learning with Ray
35mOperating a Production-Scale Physical AI Data Platform with Ray