Optimizing Large-Scale Model Training with Ray Compiled Graphs
summary
Ray Compiled Graphs create a static Ray-task graph with native GPU‑GPU NCCL communication and precompiled execution paths, letting different processes run different programs so multimodal CLIP (1.8B vision, 700M text) places vision on three A100/A800 GPUs and text on a single L4. Benchmarks show 23% raw throughput gain, up to 17x text‑GPU utilization and 64% vision uplift, and a configuration that yields ~40% throughput‑per‑dollar improvement.
AI-written summary · about
chapters (10)
- 0:00Introduction and motivation
- 3:20Ray compiled graphs overview
- 6:08Multimodal training with Ray
- 13:43Benchmark results for multimodal
- 14:04Pipeline parallelism basics
- 17:18Implementing pipeline schedules
- 21:18Zero bubble pipeline schedule
- 23:05Benchmark results for pipeline
- 24:09Summary and future work
- 25:12Q&A session
related talks
28mScaling Multimodal Data and Reinforcement Learning with Ray
26mAdapting Ray to Scale RL Training for Nemotron
19mCollective-on-Ray: High-performance Collective Communication for Distributed Machine Learning on Ray
27mEvolving and Scaling Ray for the Next Generation of AI Workloads
17mFrom Agents to Action: Scaling Agentic AI with Ray on Intel Xeon
22mRay, a Unified Distributed Framework for the Modern AI Stack
11mKeynote Q&A
11mvLLM & Ray Updates