Graph Based Pipeline Parallelism
summary
Compiles stage code to reusable FX graphs and runs explicit schedules via a schedule IR now upstreamed in torch.distributed, integrated with TorchTitan and AutoParallel and tested on 64 H100 GPUs (8 nodes × 8). Claims: no manual thread/stream hacks, compatibility with torch.compile and activation checkpointing, and graph passes that produce graph-based first performance. Measured speedups vs eager: 70% speedup for dual-pipe, 60% for zero-bubble, 50% for 1F1B; without compilation dual-pipe adds ~17% over 1F1B.
AI-written summary · about
chapters (10)
- 0:00Graph-based pipeline parallelism
- 1:24Manual schedule challenges
- 2:48Solution overview
- 3:51Building intuition backward
- 4:53Schedule IR for pipelines
- 6:18Graph partitioning workflow
- 7:42Runtime with graph callables
- 8:27DeepSeek performance results
- 9:10Graph pass benefits
- 9:31Customizing schedules
related talks
16mRevisiting Pipeline Parallelism for LLM Serving
22mCustom Pipeline Parallelism for Frontier-Scale Model Serving
14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...
15mPipeThreader: Software-Defined Pipelining for Efficient DNN Execution
11mLightning Talk: Tensor and 2D Parallelism
23mBlazing Fast GenAI Inference With Torch.compile
14mnnScaler: Constraint-Guided Parallelization Plan Generation for Deep Learning Training
29mKeynote