FlashNeuron: SSD-Enabled Large-Batch Training of Very Deep Neural Networks
summary
System moves intermediate activations from GPU HBM to NVMe SSDs and schedules transfers so they finish within the forward pass, using an offloading scheduler, a memory manager and peer-to-peer direct storage access (P2P DSA) layer. Prototype on NVIDIA Tesla V100 (16 GB) shows maximum batch size grows by 12.4x–14.0x and training throughput improves up to 37.8% (average 30.3%), with minimal CPU interference and reported ~35.3x cost efficiency per GB.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:39Rise of DNNS
- 1:30Data Reuse in DNN Training
- 2:21Memory Capacity Wall in DNN Training
- 2:47Overcoming GPU Memory Capacity Wall
- 3:41Our Proposal: FlashNeuron
- 4:50System Overview
- 5:52Offloading Scheduler: Phase 1
- 8:49Peer-to-peer Direct Storage Access (P2P-DSA)
- 10:41Evaluation: Co-locating Bandwidth-Intensive Tasks on CPU
- 11:14Conclusion
related talks
26mDeepSpeed: Efficient Training Scalability for Deep Learning
19mDeepSpeed – Efficient Training Scalability for Deep Learning Models
24mManipulating Data in an Artificial Intelligence World
16mCognitive SSD: A Deep Learning Engine for In-Storage Data Retrieval
24mBLAS-on-flash: An Efficient Alternative for Large Scale ML Training and Inference?
29mEnabling Lightweight, High-Performance FSDP With NVIDIA GPU
26mEfficiently serving LLMs at scale.
01h 12m30 Golden Rules of Deep Learning Performance