HeteCCL: Synthesizing Near-Optimal Collective Communication Schedules for Heterogeneous
summary
HeteCCL generates near‑optimal collective schedules by quantizing data into step‑level chunks, modeling topology and link bandwidth precisely, and casting scheduling as a maximum parallel transfer problem on a weighted directed graph solved with SMT formulas and counterexample‑guided inductive synthesis. Implementation synthesizes schedules two orders of magnitude faster than prior synthesizers, achieves up to 2.8×, 4.4×, and 2.6× higher bandwidth versus NCCL, TACCL, and TE‑CCL on 32‑GPU H20/V100 testbeds, and speeds end‑to‑end training by 23%–37%.
AI-written summary · about
chapters (11)
- 0:00Introduction to HeteCCL
- 1:15Limitations of fixed schedules
- 2:55Challenges in synthesis
- 4:09Impact of heterogeneity
- 6:28Pipeline and model issues
- 8:00SMT problem formulation
- 9:53Chunking abstraction
- 10:50Search space reduction
- 11:22Inductive synthesis approach
- 12:27Evaluation and results
- 13:38Conclusion
related talks
15mAutoCCL: Automated Collective Communication Tuning for Accelerating Distributed and...
15mTACCL: Guiding Collective Algorithm Synthesis using Communication Sketches
14mEfficient Direct-Connect Topologies for Collective Communications
15mACCL+: an FPGA-Based Collective Engine for Distributed Applications
22mBetter Together: Jointly Optimizing ML Collective Scheduling and Execution Planning...
01h 00mHeterogeneous Programming in C++ with SYCL 2020
22mCXL-ANNS: Software-Hardware Collaborative Memory Disaggregation and Computation...
20mBoosting Training and Inference Performance via Topology-Aware Scheduling of Heterogeneous Resources