Better Together: Jointly Optimizing ML Collective Scheduling and Execution Planning...
summary
SYNDICATE is a system that jointly optimizes scheduling and execution planning for ML collectives by introducing a motif abstraction to break collectives into fine‑grained pieces and using a centralized control plane with probabilistic search as the joint optimizer. It enforcer maps motif schedules to a data plane and is evaluated on a 16‑node, 128 GPU cluster (NVSwitch + RoCE), achieving 21–74% throughput gains on large recommendation models while incurring a small overhead on simple all‑reduce vision workloads.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:52A brief history of network optimizations in distributed training
- 7:17Evolving networking infrastructure
- 9:29We need flexible execution planning
- 11:12A case for joint optimization
- 11:58Joint optimization improvement
- 15:11Syndicate: Control Data Plane Architecture
- 16:34Syndicate: Joint Optimization with Motifs
- 17:37Evaluations
- 18:44Syndicate performance
- 20:48Conclusion
related talks
16mCASSINI: Network-Aware Job Scheduling in Machine Learning Clusters
16mIntelligent Resource Scheduling for Co-located Latency-critical Services: A Multi-Model
20mBoosting Training and Inference Performance via Topology-Aware Scheduling of Heterogeneous Resources
05mScheduling at the Edge of Reason: Multi-Cluster AI & OCM
11mScaling Distributed Machine Learning with In-Network Aggregation
15mEffectively Scheduling Computational Graphs of Deep Neural Networks toward Their...
14mPollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...