Ray Train on Anyscale: Observability and Reliability at Scale
summary
Ray Train on Anyscale adds data ingest observability (Ray Data) to show that >30% of an average batch cycle can be blocked by a slow collate stage, pinning the bottleneck to a specific transform; it polls NVIDIA RAZ every ~10 minutes for proactive hang detection (before PyTorch’s ~30‑minute timeout) and snapshots RAZ reports, per‑rank stacks, Flight Recorder and NVIDIA SMI; and it uses JIT checkpointing on 4 H100 spot nodes to cut recompute from 672 steps to 4 and lower cost ~60% vs on‑demand (≈10% better than periodic).
AI-written summary · about
chapters (6)
- 0:00Introduction
- 0:29Common failures in distributed training
- 4:49Data starvation and observability
- 12:48Training hangs with nickel
- 17:46Node preemption handling
- 20:15Summary and Q&A
related talks
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
20mThe Quick Journey to Using Ray: How We Implement Ray and Anyscale to Speed up our ML Processes
33mOvercoming Distributed ML Challenges with Ray Train
22mLarge Scale Data Loading and Data Preprocessing with Ray
30mHow to Scale Unstructured Data Processing with Ray Data
32mAnyscale's Ray Data: Revolutionizing Batch Inference
28mScaling Multimodal Data and Reinforcement Learning with Ray
13mRay and Anyscale: An Optimization Journey