Big Data Processing with GPUs
summary
PayPal reports 70% cost reduction by moving a Spark 2 pipeline (140 nodes, 2.0 h) to Spark 3 with GPU acceleration (30 nodes, 1.3 h) using Spark RAPIDS and two Tesla T4 GPUs per node; common heavy-aggregation and large-shuffle jobs show the biggest gains. Key operational fixes: enable large partitions, tune Spark 3 for big partitions, increase disks per node from 4 to 8 to avoid "no space left on device," and consider L4 GPUs for better perf/cost on scaled GPU clusters.
AI-written summary · about
chapters (6)
- 0:00Introduction to big data GPUs
- 3:41GPU vs CPU fundamentals
- 5:11Spark Rapids accelerator overview
- 8:04Real-world production test
- 11:43Cost comparison results
- 12:46Tips for GPU adoption
related talks
30mSpeed Up Your Spark Data Processing with a GPU
41mAccelerating Data Science with RAPIDS
19mSpark vs. Accelerated Compute: A Balanced Look at Data Processing Strategies
38mRAPIDS- Open Source GPU Data Science
26mLarge Scale Graph Processing & Machine Learning Algorithms
42mMulti-GPU Accelerated Processing of Time-Series Data of Huge Academic
39mAccelerating Data Science with RAPIDS
01h 22mHigh-Performance Data Science at Scale with RAPIDS, Dask, and GPUs