DShuffle: DPU-Optimized Shuffle Framework for Large-scale Data Processing
summary
DShuffle fully offloads Spark shuffle onto Bloomfield 3 DPUs using DPA-accelerated serialization and DPU direct spilling via PCIe P2P, and organizes work into a fine-grain pipeline of serializer, shuffle, and spill workers. On a two-node 100 Gb testbed with NVMe SSDs it reports 1.33–1.5× job speedups on shuffle-heavy HiBench workloads, increases network and disk utilization by ~3× and ~2×, reduces host CPU use by about eight cores and cuts GC-related CPU time by ~20%.
AI-written summary · about
chapters (14)
- 0:00DShuffle framework intro
- 2:50CPU overhead in shuffles
- 4:17DPU shuffle offloading
- 7:07DShuffle design
- 8:30DShuffle techniques
- 9:34DPA serialization
- 10:40DPU direct spilling
- 12:05Pipeline parallelism
- 13:10Experimental setup
- 14:13Performance improvements
- 14:56Resource utilization
- 16:22Scalability results
- 17:06High concurrency perf
- 17:49Summary
related talks
14mPD3: Prefetching Data with DPUs for Disaggregated Memory
16mMinFlow: High-performance and Cost-efficient Data Passing for I/O-intensive Stateful...
19mCluster Optimization: Harnessing DPUs for Ambient Mesh Workloads
14mAccelerating PyTorch FSDP Via Overlapping Collectives With In...
10mPyTorch Distributed Data Parallel (DDP)
18mGPU-Disaggregated Serving for Deep Learning Recommendation Models at Scale
10mDiSh: Dynamic Shell-Script Distribution
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads