FlexPipe: Maximizing Training Efficiency for Transformer-based Models with...
summary
FlexPipe is a middleware that dynamically resizes pipeline parallelism per iteration, adding or removing PP stages without stalling via a live flexibility mechanism that overlaps initialization, communication, and computation. It formulates the Flexible Memory Optimization Problem and solves it with the HBISA heuristic to choose stage-to-device mappings and trigger conditions. Implemented on 8 servers × 4 A100 GPUs, experiments on BERT/GPT variants show about 1.25× training throughput vs state-of-the-art and up to ~40% gain over ZeroBubble.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:39Transformer training challenges
- 2:25Pipeline parallelism limits
- 3:49FlexPipe's core insight
- 4:56FlexPipe architecture
- 7:03Flexible Memory Optimization
- 8:54Evaluation and performance
- 11:27Overhead and ablation studies
- 13:13Summary
- 13:54Q&A
related talks
14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...
17mAttack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training
20mHetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through
21mEnvPipe: Performance-preserving DNN Training Framework for Saving Energy
15mPipeThreader: Software-Defined Pipelining for Efficient DNN Execution
16mRevisiting Pipeline Parallelism for LLM Serving
09mFlexAttention + FlashAttention-4: Fast and Flexible
10mGraph Based Pipeline Parallelism