PipeThreader: Software-Defined Pipelining for Efficient DNN Execution
summary
A compiler shifts pipeline scheduling from hardware to software, exposing tile-level tasks and heterogeneous units to generate efficient DNN pipelines. It defines sTask-graph nodes mapped to SEU/EU hardware, and new primitives (append/wait/propagate) to place and order work. PipeThreader automatically rediscovers FlashAttention-style pipelining, achieves up to 2.18× speedup on H100 versus handcrafted kernels, finds faster schedules for Mamba2 and Llama3, and targets AMD Mi300X; code is open-sourced.
AI-written summary · about
chapters (7)
- 0:00PipeThreader introduction
- 3:10Handcrafted DNN kernel challenges
- 7:24Software-defined pipelining approach
- 10:17PEPThreader abstractions
- 12:46Automated scheduling
- 13:26Performance gains
- 14:31Conclusion
related talks
20mHetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through
16mRevisiting Pipeline Parallelism for LLM Serving
14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...
17mPPipe: Efficient Video Analytics Serving on Heterogeneous GPU Clusters via...
10mGraph Based Pipeline Parallelism
21mEnvPipe: Performance-preserving DNN Training Framework for Saving Energy
17mAttack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training
35mNNStreamer: Efficient and Agile On-Device AI Application Development with Neural-Net Pipelines