An Initial Look at Deep Learning IO Performance
summary
Benchmarks use ImageNet (1.3M images, ~160 GB) and TFRecords (about 1,000 files ~1.16 GB each) on Lambda Labs instances with 8 V100 GPUs and stock setups. Synthetic runs hit >10,000 images/sec but real reads top around ~8,000 images/sec after saturating ~28 CPU IO threads and ~1–1.3 GB/s client read. JPEG decompression is CPU-bound, so converting to TFRecords or offloading with DALI (GPU-side decoding) removes metadata pressure; Ceph RBD/CephFS can work with tuning and RBD shows 128KB random read ~75 GB/s on a 60-OSD NVMe cluster.
AI-written summary · about
chapters (16)
- 0:00Introduction and welcome
- 3:07Why AI and storage matter
- 5:57Deep learning IO challenges
- 9:09Testing with synthetic data
- 11:16Real-world dataset ImageNet
- 13:23TFRecords and performance gains
- 15:54Analyzing IO bottlenecks
- 17:19Implications for Ceph storage
- 21:35RBD vs RadosGW vs CephFS
- 25:27S3 and emerging protocols
- 26:50Nvidia DALI and future optimizations
- 28:34Ceph's potential in AI workloads
- 31:22TFRecords conversion trade-offs
- 33:34Alternative data formats like Zarr
- 35:08Future workflows and AI research
- 35:51GPU Direct Storage and next steps
related talks
01h 12m30 Golden Rules of Deep Learning Performance
34mDeep Learning at Scale
44mDeep Learning in Production
17mAll AI Roads Lead to Distribution
14mSoftware 2.0 Needs Data 2.0: A New Way of Storing and Managing Data for Efficient Deep Learning
24mManipulating Data in an Artificial Intelligence World
04mProcessing 100+ million images per month with deep learning
42mDeep Learning is Like Water (it's Everywhere!)