Dataloader Design for PyTorch
summary
PyTorch DataLoader now lets datasets return whole batches and supports iterable streams so data loading moves from one-index-per-call to either batch-style dataset or IterableDataset. Samplers can emit lists of indices so the dataloader no longer must assemble batches one sample at a time, cutting round trips when reading from databases and enabling efficient RL/streaming workloads. The change is implemented in recent releases and preserves existing collation while giving per-batch control (filtering, timeouts, marking indices).
AI-written summary · about
chapters (6)
- 0:00Introduction to data loading
- 0:39Core data loading components
- 1:59Batch loading improvements
- 3:09Implementing batch loading
- 4:39Iterable datasets
- 5:45Summary of updates
related talks
24mThe Evolving Landscape of Dataloading
28mPer-epoch Shuffling Data Loader: Mix It Up As You Train!
22mLarge Scale Data Loading and Data Preprocessing with Ray
28mFault Tolerance Training at Pinterest
08mPyTorch Distributed: Towards Large Scale Training
18mData Preprocessing for ML at Production Scale
08mTorchText
19mIncreasing the Impact of AI Through Better Software (Pytorch Updates)