Ray and Anyscale: An Optimization Journey
summary
Delivers a concrete pipeline that turns a trained model into a scalable REST endpoint using Ray Serve on Anyscale, Elasticsearch and Locust for load testing, moving from ~1.5 RPS to ~5–6 RPS on one instance by introducing ray object store caching and then to much higher throughput with an actor-based cache that preserves objects across nodes, reaching a reported 17× effectiveness with four replicas and simple scale-up hooks that work in the demo.
AI-written summary · about
chapters (10)
- 0:00Introduction and overview
- 1:04Team and problem statement
- 2:29Initial architecture and first attempt
- 3:31First implementation with Ray Serve
- 5:17Identifying inefficiency and caching need
- 6:24Using Ray Object Store for caching
- 8:13Scaling up with Reactor Model
- 9:19Implementing Actor Model
- 10:43Live scaling demonstration
- 12:29Final results and conclusion
related talks
30mScaling Machine Learning at Discord: Our Journey with Ray and Anyscale
31mManaging Large-Scale Ray Deployments: Cloud, On-Prem, and Kubernetes
16mFrom Self-Managed Ray to Anyscale Services: Migrating ML Serving
31mFaster Model Serving with Ray and Anyscale
23mIntroducing Ray Serve: Scalable and Programmable ML Serving Framework
30mHow to Scale Unstructured Data Processing with Ray Data
32mAnyscale's Ray Data: Revolutionizing Batch Inference
21mRay Train on Anyscale: Observability and Reliability at Scale