How to accelerate Apache Spark pipelines on Amazon EMR with RAPIDS
summary
RAPIDS Accelerator is a Java plugin that hooks into Spark 3 via a jar (Maven Central or built into EMR), replaces CPU execs with GPU execs through JNI into RAPIDS libraries (cuDF, IO, memory manager) and falls back to CPU for unsupported ops. Deployment on EMR requires three steps: enable the plugin, configure YARN cgroups for GPUs, and set Spark configs. Benchmarks use an adapted TPC-DS (3 TB Parquet on g5.8xlarge) and show 2–5x speed up / 30% cost savings for the top queries; qualification runs Spark 2/3 event logs and reports recommended jobs with conservative estimates and is available as pip install spark-rapids-user-tools; no code change required.
AI-written summary · about
chapters (7)
- 0:00The data processing challenge
- 2:23Introduction to Rapids
- 4:11How the accelerator works
- 5:15Deep dive into integration
- 8:24Benchmark performance
- 11:57Qualification and tooling
- 14:47Customer success and resources
related talks
30mSpeed Up Your Spark Data Processing with a GPU
39mAccelerating Data Science with RAPIDS
01h 36mRAPIDS: GPU-Accelerated Data Science for PyData Users
14mBig Data Processing with GPUs
11mScaling R with Spark
01h 03mWorkshop: Intro to GPU data science with RAPIDS
28mInnovations in AWS analytics: Data processing
52mSolving Analytical Problems using Apache Spark