A heuristic algorithm for periodic memory allocation with little fragmentation to train(…)
summary
Claims an offline simulated-annealing heuristic that reorders allocation topological ordering to cut PyTorch caching fragmentation from 29.5% wasted by PyTorch caching allocator down to 0.4% fragmentation (first-fit gives 5.3%), evaluates candidates in O(logN) amortized time, traces one training iteration then applies the plan, handles 2k–9k allocations from vision and language training and random tests, and reports near-zero fragmentation with preserved training throughput.
AI-written summary · about
chapters (12)
- 0:00Memory fragmentation in training
- 1:02PyTorch allocator fragmentation
- 2:50Recomputation impact on memory
- 4:58Why fragmentation matters
- 5:18Offline memory planning
- 7:07Dynamic storage allocation
- 9:19Simulated annealing basics
- 11:55Optimizing simulated annealing
- 15:56Experimental results
- 18:05Training throughput comparison
- 19:10Conclusion and future
- 19:35Q&A
related talks
18mTierTrain: Proactive Memory Tiering for CPU-Based DNN Training
05mPlatform-Agnostic Lightweight Deep Learning for Garbage Collection...
15mRefurbish Your Training Data: Reusing Partially Augmented Samples...
28mMaximize GPU Utilization for Model Training
16mtorchtune: Easy and Accessible Finetuning in Native PyTorch
15m3L-Cache: Low Overhead and Precise Learning-based Eviction Policy for Caches
20mBoosting Training and Inference Performance via Topology-Aware Scheduling of Heterogeneous Resources
18mSHADE: Enable Fundamental Cacheability for Distributed Deep Learning Training