Overcoming Distributed ML Challenges with Ray Train
summary
Ray Train is a Python library that sets up distributed training from one line of scaling config, launches workers, and configures framework-specific environments (PyTorch DDP, FSDP, DeepSpeed). It pairs with Ray Data for sharded streaming ingestion and last‑mile preprocessing, supports checkpointing and trainer.Restore for fault tolerance and elastic training to continue on fewer GPUs. A Stable Diffusion v2 pretraining case uses Ray Data offline/online preprocessing to free encoder GPU time, yielding about a 30% throughput gain and an additional >300% boost after model-level optimizations, while trimming cost ~10%.
AI-written summary · about
chapters (9)
- 0:00Introduction and trends
- 1:56Distributed training hurdles
- 3:26Getting started with Ray Train
- 7:15Workflow integration
- 8:29Distributed observability
- 11:50Fault tolerance and elasticity
- 16:37Large-scale data ingestion
- 22:14Stable diffusion case study
- 27:28Conclusion and Q&A
related talks
22mLarge Scale Data Loading and Data Preprocessing with Ray
26mRayDP: Build Large-scale End-to-end Data Analytics and AI Pipelines Using Spark and Ray
22mRay, a Unified Distributed Framework for the Modern AI Stack
28mScaling Multimodal Data and Reinforcement Learning with Ray
35mOperating a Production-Scale Physical AI Data Platform with Ray
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
24mRay: A System for High-performance, Distributed ML Applications
26mData Processing on Ray