Collective-on-Ray: High-performance Collective Communication for Distributed Machine Learning on Ray
summary
Ray-native Python collective primitives run on top of Python bindings to NCCL and Gloo and implement all-reduce, broadcast, point-to-point and declarative collectives for CPU and GPU arrays. Benchmarks report up to 1000x lower latency on single-node GPU and 10–100x speedups across multi-node clusters versus Ray RPC/object-store assembly. The project builds a training library (parameter-server and pipeline-parallel strategies), but collectives bypass the object store (no automatic fault tolerance) and require every participant to invoke calls exactly once, risking deadlocks.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:44Distributed Communication
- 1:17Ray Simplifies P2P Communication A Lot!
- 1:51Collective Communication Patterns
- 2:50A More Complex Case: Ring AllReduce
- 3:49Introducing Collective Primitives to Ray
- 4:30Pre-cap: Advantages Programmatically simple
- 5:14Walk-through Example: Data-parallel Training
- 6:13Training Workers as GPU Actors
- 7:08Assembling AllReduce Using ray.get()
- 8:57Overheads
- 10:02Using Ray Collective Primitive APIs
- 10:56Behind the Scene: CCL Backends
- 11:50Microbenchmarks #1
- 12:52Case: Spacy NER + Parameter Server
- 13:52Supporting Matrix
- 14:53Limitations
- 17:14Future Steps: Distributed ML on Ray
- 17:43Take-aways
related talks
24mRay: A System for High-performance, Distributed ML Applications
15mRay: A System for Distributed Applications
01h 31mIntroduction to Ray for distributed and machine learning applications in Python
26mRay: A System for High-performance, Distributed Python Applications
15mRay: A Distributed Compute Engine for AI
42mRay: A distributed system for emerging AI applications
22mRay, a Unified Distributed Framework for the Modern AI Stack
11mKeynote Q&A