Speed Up Your Spark Data Processing with a GPU
summary
NVIDIA’s Spark RAPIDS plugin enables running Spark DataFrame work on GPUs (usable on clouds such as AWS EMR) by installing the right drivers, two JARs, and enabling the plugin. Real-world tests show typical 2–3× speedups and ~2× cost savings for ETL-heavy jobs (up to ~4× on small examples), but ~8% of jobs may fail or fall back to CPU; production use requires Yarn/GPU allocation, per-GPU memory planning (e.g. T4 ≈16GB), large input chunk sizes (hundreds of MB), substantial tuning (~200 parameters), and has incompatibilities with decimal types and nested structs.
AI-written summary · about
chapters (9)
- 0:00Start
- 0:57RAPIDS ecosystem
- 5:37Code change requirements
- 5:54Where's the magic ?
- 14:22Cluster configuration is more complex
- 16:36Some cluster job results
- 17:54EMR 6.3 Benchmark profile
- 20:15Some job/queries show impressive improvements!
- 24:21Efficient GPU scheduling is hard in the cloud
related talks
19mHow to accelerate Apache Spark pipelines on Amazon EMR with RAPIDS
14mBig Data Processing with GPUs
38mRAPIDS- Open Source GPU Data Science
41mAccelerating Data Science with RAPIDS
39mAccelerating Data Science with RAPIDS
19mSpark vs. Accelerated Compute: A Balanced Look at Data Processing Strategies
01h 22mHigh-Performance Data Science at Scale with RAPIDS, Dask, and GPUs
28mInnovations in AWS analytics: Data processing