Per-epoch Shuffling Data Loader: Mix It Up As You Train!
summary
Ray-based per-epoch shuffling data loader yields globally shuffled GPU batches to dozens of trainers via an iterable dataset API, using a mapper→reducer all-to-all shuffle, queuing with backpressure, prefetching and pipelined per-epoch shuffles (default two concurrent epoch shuffles). Benchmarks show ~5.5M records/sec across nodes and shuffled batches reach trainers in ~100 ms after the first epoch. Integrates with Horovod-on-Ray, reads from S3/HDFS, and aims to avoid PyTorch/Petastorm shuffle latencies.
AI-written summary · about
chapters (7)
- 0:00Introduction to talk
- 1:25What is a shuffling data loader
- 7:01Challenges in shuffling data loaders
- 12:35Why Ray is a good fit
- 17:08Shuffling data loader design
- 25:13Benchmark results and benefits
- 27:41Key takeaways and conclusion
related talks
33mOvercoming Distributed ML Challenges with Ray Train
30mScaling Machine Learning at Discord: Our Journey with Ray and Anyscale
32mAnyscale's Ray Data: Revolutionizing Batch Inference
27mPinterest's ML Evolution: Distributed Training with Ray
29mRay Data on GPU - Breaking Preprocessing Bottleneck for Model Training
24mThe Evolving Landscape of Dataloading
28mFault Tolerance Training at Pinterest
36mRay Data: What's New - GPU Shuffle, Unity Catalog, DataSourceV2