AutoCCL: Automated Collective Communication Tuning for Accelerating Distributed and...
summary
AutoCCL automates low-level tuning of NCCL to cut communication overhead: it prunes NCCL's 158 knobs to 28 parameters, partitions the huge search space with a divide-and-conquer approach and uses coordinate-descent within subspaces, and measures real communication under computation using online tuning hidden in early iterations. Evaluations on 16- and 32-GPU A40 clusters report microbenchmark speedups of 1.24–1.29× vs NCCL and 1.15–1.22× vs a SOTA tuner (up to 1.80×/1.49× with overlap) and 1.07–1.32× end-to-end model gains.
AI-written summary · about
chapters (12)
- 0:00AutoCCL and collective comms
- 1:22Role of collective comms in training
- 2:47Limitations of existing libs
- 3:53Challenges in tuning comms
- 5:17Reducing tuning parameter space
- 6:40Navigating parameter space
- 8:07Handling computational interference
- 9:34End-to-end tuning with low overhead
- 10:36Comparison with existing approaches
- 12:04Evaluation on real workloads
- 13:51Future directions and open-source
- 14:38Q&A
related talks
14mHeteCCL: Synthesizing Near-Optimal Collective Communication Schedules for Heterogeneous
01h 17m📚 Advanced NCCL Tutorial: Beyond Collectives
15mTACCL: Guiding Collective Algorithm Synthesis using Communication Sketches
15mACCL+: an FPGA-Based Collective Engine for Distributed Applications
19mCausalTuner: Feature-Aware Causal Guidance for Compiler Auto-tuning
10mAuto-tuning vllm
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
17mSelfTune: Tuning Cluster Managers