From Self-Managed Ray to Anyscale Services: Migrating ML Serving
summary
Coinbase migrates production serving from self‑managed Ray Serve on EKS to Anyscale managed Ray, moving 50+ models and 1 billion inferences per day into production. The migration replaces homogeneous GPU-heavy pools with per‑domain clusters (about 20 managed clusters, 10 instance types, up to 500 nodes) and eliminates noisy‑neighbor outages by isolating workloads. Deploys shift from 30–60 minute manual rollouts to under five minutes deploys driven by codified YAML, MLflow artifacts, shadow parity checks and GitHub Actions.
AI-written summary · about
chapters (8)
- 0:00Introduction to ML serving at Coinbase
- 2:40Legacy serving stack challenges
- 4:30Shared Ray cluster limitations
- 6:58Why switch to Anyscale
- 8:01Migration process to Anyscale
- 10:07Benefits of Anyscale clusters
- 12:33Operational improvements
- 13:35Next steps and ownership
related talks
20mThe Quick Journey to Using Ray: How We Implement Ray and Anyscale to Speed up our ML Processes
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
31mManaging Large-Scale Ray Deployments: Cloud, On-Prem, and Kubernetes
31mFaster Model Serving with Ray and Anyscale
13mRay and Anyscale: An Optimization Journey
29mScaling ML Pipelines for Autonomous Driving with Ray
35mCoinbase's ML Training Evolution: From Sagemaker to Ray
35mReddit's ML Evolution: Scaling with Ray and KubeRay