An Open Source Post-Training Stack: Kubernetes + Ray + PyTorch + vLLM
summary
An open source stack using Kubernetes + Ray + PyTorch + vLLM runs post‑training workloads by letting Kubernetes provision nodes, Ray orchestrate heterogeneous CPU/GPU pools, PyTorch execute model kernels, and vLLM handle low‑latency inference; case studies from Pinterest, Uber, and Roblox show this reduces tail latency and operational overhead. Ray + vLLM integration lets teams separate prefill and decode pools for predictable performance across sharded ranks, and Ray is now donated Ray to the PyTorch Foundation, signaling community stewardship.
AI-written summary · about
chapters (10)
- 0:00Open source AI compute stack
- 1:02Ray joins PyTorch Foundation
- 2:25Ray for generative AI
- 3:28SQL to GPU inference
- 6:37Reinforcement learning challenges
- 11:32AI workload software stack
- 14:02Ray and vLLM for inference
- 16:48Anyscale’s role in Ray
- 17:49Ray Summit and community
- 18:31Q&A
related talks
16mKeynote: An Open-Source Stack for AI Compute: Kubernetes + Ray + PyTorch + vLLM
28mScaling Multimodal Data and Reinforcement Learning with Ray
32mAn Overview of CloudKitchens's Ray-Powered ML Platform
23mCost-Effective RL Post-Training with Ray and AWS Trainium
21mEnd-to-End AutoML with Ludwig on Ray
28mEfficient Deep Learning Training with Ludwig AutoML, Ray, and Nodeless Kubernetes
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
33mOvercoming Distributed ML Challenges with Ray Train