Spark vs. Accelerated Compute: A Balanced Look at Data Processing Strategies
summary
Kroger runs 2,700 stores and $150B in annual revenue, so efficiency matters. Benchmarks show concrete wins: 1B-row sort: Spark 112s, Polars CPU 15s, Polars GPU 5s, cuDF 0.86s; join test: Spark 145s, Polars 26s, cuDF ~200× faster; Polars, cuDF (GPU), Polars-on-GPU, Pandas-compatible APIs, Spark Rapids and A100/H200 hardware are the actual tools compared. Guidance: use GPUs for gigabytes/billions of rows and low‑latency or matrix/graph work; keep Spark for terabyte+ network‑bound, orchestration, and resilience.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:57Scale-out vs scale-in
- 3:24Batch to real-time
- 5:09Spark vs accelerated stacks
- 7:13Billion-row sort benchmark
- 8:41Needle in haystack tests
- 10:07Join operations benefits
- 11:54GPU memory limits
- 13:19Spark or accelerated choice
- 14:43Accelerated use cases
- 15:45Spark Rapids hybrid
- 16:26Speed as capability
related talks
37mSpark, Dask, DuckDB, and Polars: Benchmarks
30mSpeed Up Your Spark Data Processing with a GPU
31mPolars, the End of Spark?
14mBig Data Processing with GPUs
39mAccelerating Data Science with RAPIDS
41mAccelerating Data Science with RAPIDS
38mRAPIDS- Open Source GPU Data Science
28mInnovations in AWS analytics: Data processing