Reddit's ML Evolution: Scaling with Ray and KubeRay
summary
Reddit moves its training and serving stack onto Ray and KubeRay, building Ray workspaces with a persistent Jupyter head node, a Git-tagged Docker release flow, and a control plane that includes a Gazette API and Kubernetes controller for Ray jobs. That tooling yields a claimed 6x training speedup, lets a safety team train 5–10 extra models monthly, and drives 10x throughput at 10x lower cost for async GPU inference with dynamic batching.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:41Why Ray and KubeRay
- 8:03How Reddit uses Ray
- 13:51Shared Code
- 15:51Ray CLI
- 17:23Production
- 18:32Workflow
- 19:53TLDR
- 22:00Ray Serve
- 22:39KubeRay Serve
- 23:42Model Server Deployment
- 24:02Dynamic Batching
- 24:31Expressive Handler
- 25:12Open Source ML
- 25:45LLYM Playground
- 26:04Results
- 26:57Questions
related talks
28mScaling Multimodal Data and Reinforcement Learning with Ray
44mPanel: ML Platform on Ray
16mScaling Massive Transformer Training with KubeRay
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
31mHosting AI models with Ray and GKE
28mRay on Public-Cloud k8s - experiments, lessons & best practices