nnScaler: Constraint-Guided Parallelization Plan Generation for Deep Learning Training
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
A plan-generation framework implements three primitives op-trans, op-assign, op-order plus constraint filters so experts compose custom parallelization spaces instead of fixed 3D presets; the system generates executable PyTorch code, inserts communications, and checks graph legality. It discovers new schemes such as kard (reduces peak memory, 1.5× on SwinTransformer) and interlaced pipeline (2.5× on T5, finds AlphaFold2 plan using <10% search time). End-to-end runs scale to 512 GPUs and a 92B-parameter model and report up to 3.5× speedups versus DeepSpeed/Megatron-LM/Alpa.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:09Parallel training plan
- 2:16Current parallelization practices
- 3:24NN scaler design
- 4:27Space construction primitives
- 5:34Constraints to reduce search
- 6:38Two-step search policy
- 7:41Materializing the plan
- 9:05New plans discovered
- 11:35Performance evaluation
- 12:20Real-world applications
- 13:48Summary
related talks
26mDeepSpeed: Efficient Training Scalability for Deep Learning
10mGraph Based Pipeline Parallelism
34mPanel Discussion: Training and Inference at Planet Scale
01h 03mScaling up machine learning without tears (and what do programming languages ha...
15mEffectively Scheduling Computational Graphs of Deep Neural Networks toward Their...
15mAlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
24mTraining large scale models using PyTorch
16mGalvatron: An Automatic Distributed Training System for Efficient Large...