CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training
summary
CrossPipe explicitly models cross-datacenter latency and bandwidth with an alpha–beta cost model and produces solver-based optimal or greedy near-optimal pipeline schedules that keep identical peak memory. It implements scheduling and communication orchestration inside Megatron LM, injects synthetic delays on a Swiss supercomputer to emulate cross-DC links, and evaluates on LLaMA-like models. Results show up to 33.6% training time reduction versus traditional schedules while the Megatron LM integration runs and scales across 2–4 datacenters.
AI-written summary · about
chapters (10)
- 0:00Cross-datacenter LLM training
- 1:42Cross-datacenter parallelism challenges
- 3:50Pipeline parallelism dependencies
- 6:00Pipeline bubbles in training
- 7:23Optimal schedule construction
- 8:06Greedy schedule generation
- 9:09Pipeline vs data parallelism
- 11:16CrossPipe in Megatron LM
- 14:06CrossPipe evaluation results
- 17:18Summary and release
related talks
17mAttack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training
20mHetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through
14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...
15mPipeThreader: Software-Defined Pipelining for Efficient DNN Execution
22mCustom Pipeline Parallelism for Frontier-Scale Model Serving
47mDesigning High-Performance Scalable Middleware for HPC, AI, & Data Science in Exascale Systems & ...
16mRevisiting Pipeline Parallelism for LLM Serving
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective