Is Spark still relevant? Multi-node CPU and single-node GPU workloads..
summary
Compares Spark Dask and RAPIDS across exploration modeling prediction and dashboarding and concludes Spark remains relevant because enterprises prioritize maturity and vendor SLAs over raw benchmarks. Shows RAPIDS gives huge single‑node GPU speed but lacks mature distributed tooling and consultancies; Dask runs well on Kubernetes with dask‑gateway but misses a production SQL compiler; Spark wins for job scheduling (YARN), broad vendor ecosystem and training availability. Demonstrates a planned bake‑off is infeasible without more infrastructure so the comparison is pragmatic not purely performance‑driven.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 3:12Spark relevance today
- 8:09Spark vs Dask vs RAPIDS
- 19:22RAPIDS and Dask setups
- 22:54Open-source adoption challenges
- 27:04Mitigating RAPIDS risks
- 30:32Summary and outlook
related talks
28mAdapting from Spark to Dask: what to expect
25mDistributed data workflows: PySpark vs Dask
41mAccelerating Data Science with RAPIDS
43mPySpark in Practice
46mJulia, Python, R & Scala – A Comparison From Experience
39mScaling Python for Machine Learning: Beyond Data Parallelism
25mScale Data Science by Pandas API on Spark
11mThe Evolution of Apache Spark™