Scaling Parallel Algorithms to Massive Datasets using Multi-SSD Machines
summary
A 64-core Threadripper machine outfitted with 31 PCIe SSDs and about 1/2 TB DRAM saturates at ~135 GB/s aggregate I/O; a user-space library built around io_uring and XFS reaches about 120 GB/s for sequential reads, while a reduce kernel falls to ~60 GB/s due to two memory transfers per cache line. An external-memory samplesort, implemented in ~100 lines, sorts up to 2^48 8-byte integers with ~20% slowdown versus an optimized in-memory version at large scale.
AI-written summary · about
chapters (16)
- 0:00Multi-SSD scaling research
- 1:04SSDs over RAM for large datasets
- 3:32External-memory algorithm challenges
- 5:38Building multi-SSD machine
- 9:56SSD bandwidth scaling limits
- 13:49SSD-to-RAM data transfer bottlenecks
- 16:25Reduce algorithm on multi-SSDs
- 17:49Samplesort in external-memory
- 20:18Mitigating I/O overhead with io_uring
- 23:47Multi-SSD samplesort performance
- 25:14Algorithm portability on cloud hardware
- 26:37Application performance patterns
- 28:01Modeling parallel external-memory algorithms
- 29:21Future for graph algorithms
- 30:04Hardware trends for multi-SSD
- 30:47Conclusion and open questions
related talks
01h 06mBest practices when accessing Big Data or any other data!
24mManipulating Data in an Artificial Intelligence World
32mLarge-Scale Graph Processing on Emerging Storage Devices
12mMilliSort and MilliQuery: Large-Scale Data-Intensive Computing in Milliseconds
39mUnlock the potential of your data: Optimize your data store architecture...
42mMulti-GPU Accelerated Processing of Time-Series Data of Huge Academic
05h 34mData @Scale 2024
29mOn Data Locality in Kubernetes