How Roblox Scaled Machine Learning by Leveraging Ray for Efficient Batch Inference
summary
Roblox integrates Ray (Ray Data, Ray Jobs) with KServe, Triton, Kubernetes and vLLM to run daily batch inference jobs that process up to 10+ billion tokens per job, using MVMe caching and a 5.6 GB/s model downloader to avoid 30-minute weight downloads. Moving CLIP backfills from online servers to Ray yields 10x throughput improvement and a CLIP pipeline experiment shows 58% cost savings versus an optimized online solution; platform adds a UI, SDK, custom GPU scheduler and multi‑tenant quotas that successfully run hundreds of jobs daily.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 8:35Use Cases
- 13:39Batch API
- 17:44Roblox Platform
- 18:23Architecture
- 19:34Frontend
- 20:08Job Creation Panel
- 20:42GPU Scheduler
- 22:50New System
- 23:03Authentication Authorization Flow
- 24:15Notification Client
- 24:47Roblox ML SDK
- 25:20Future Work
related talks
28mScaling Multimodal Data and Reinforcement Learning with Ray
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
21mLeveraging Ray to Enhance Machine Learning Models in Finance
28mServing LLMs for Agents at Scale with Ray + vLLM
35mOperating a Production-Scale Physical AI Data Platform with Ray
24mRay: A System for High-performance, Distributed ML Applications
30mTransforming Grab’s AI Platform with Ray
44mPanel: ML Platform on Ray