Ray and MAI-Thinking-1
summary
Microsoft AI uses Ray as the shared compute layer to build MAI-Thinking-1, running across Kubernetes and Slurm with multi‑cluster setups. They implement relay actors per cluster to fan out RPCs, keeping actor creation flat (~30s from 8k to 32k) and cutting P50 RPC time from ~50s to ~3s and further to ~100ms with a tree of relays. Fault tolerance uses runtime failover: spare racks, diagnostics actors (kernel logs, NVIDIA‑SMI/NVML), prewarmed actors, explicit node assignment and partial restarts to avoid full cold boots.
AI-written summary · about
chapters (8)
- 0:00Introduction and agenda
- 1:26Ray resource scaling
- 2:19Ray client proxy optimization
- 3:17Bottlenecks and tree scaling
- 4:49Fault tolerance and management
- 10:33Runtime failover strategies
- 17:34Ray for RL scaling and case study
- 27:50Conclusion and Q&A
related talks
23mImproving Ray for Large-scale Applications
15mRay: A Distributed Compute Engine for AI
26mUsing Ray On Large-scale Applications at Ant Group
28mScaling Multimodal Data and Reinforcement Learning with Ray
14mBuilding High Availability and Scalability Online Computing Applications on Ray
24mDistributed Black-Box Model Explanation with Ray
24mRay: A System for High-performance, Distributed ML Applications
26mRay: A General Purpose Serverless Substrate?