The Evolving Landscape of Dataloading
summary
PyTorch data loading is fragmented and increasingly the tight spot for large training jobs: attempts to ship a one‑size library paused torchdata effort in July after discovering the design can’t meet all workloads. Key pressures are scale and variance — trillions of tokens, millions/billions of images, clusters of thousands of GPUs, and hardware shifts like 600 Gbps instances. Common fixes are prefetching, parallel workers, bundling/sharding and external fetch/transform services, but long‑tail fetch latency, caching, and video decoding (orders‑of‑magnitude cost increases) remain the hard problems.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:07The data loading model
- 4:43Performance formulas
- 6:16Industry trends
- 14:29Ecosystem and fragmentation
- 16:21Challenges in abstraction
- 23:46Conclusion
related talks
24mTraining large scale models using PyTorch
07mDataloader Design for PyTorch
19mIncreasing the Impact of AI Through Better Software (Pytorch Updates)
22mLarge Scale Data Loading and Data Preprocessing with Ray
08mPyTorch Distributed: Towards Large Scale Training
28mFault Tolerance Training at Pinterest
12mThe Ultra-Scale Talk: Scaling Training to Thousands of GPUs
34mDeep Learning at Scale