DISTMM: Accelerating Distributed Multimodal Model Training
Talkat 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24) · 2024-04-16..04-18·
summary
DISTIMM implements per-submodule parallelism choices (tensor, data, hybrid), adaptive partitioning and non‑collocated placement to balance M/K/N dimensions and reduce idle GPUs, adds a batch-sync instruction and the DISTIMM‑Pipe schedule to construct full contrastive similarity matrices across microbatches, and exploits heterogeneous interconnects to cut gradient sync traffic. Prototype trains 1.1B–26B multimodal models and reports 1.32–3.27× speedup over Megatron‑LM while enabling pipeline scaling to very large global batch sizes.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 0:34Training challenges
- 10:51Proposed system components
- 11:39System design implementation
- 13:30Pipeline scheduling
- 14:28Evaluation and results
- 15:09Performance analysis
- 16:02Conclusion
related talks
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
26mAccelerating Explorations in Vision and Multimodal AI Using Pytorch...
19mML training acceleration with heterogeneous resources in Bytedance
17mAttack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training
36mAccelerating High-Performance Machine Learning at Scale i...
19mNo GPU Left Behind: Scaling Online LLM Training With...
15mOptimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation
29mTrain billion-parameter models with model parallelism on Amazon SageMaker