Distributed data workflows: PySpark vs Dask
summary
Compares PySpark and Dask with concrete benchmarks and code-level claims: Dask is pure Python (installable with pip install dask), uses DD.read_parquet and delayed objects to build lazy task graphs, and shows 1,000-row reads at 1s vs PySpark 11s, 1M rows at 1.8m vs 2.3m, filter+persist 1.6m vs 2.6m, but joins run slower on Dask (12.8m vs PySpark 5.6m). Describes a 200 ms task overhead, a diagnostic dashboard, and scaling from single-core to multi‑node clusters.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 2:20What is Spark
- 6:23What is Dask
- 14:38Benchmarks
- 17:10Which to use
- 18:38Collab notebooks
- 21:08Dask documentation
- 22:03Questions
related talks
28mAdapting from Spark to Dask: what to expect
01h 22mUsing Dask for Parallel Computing in Python
01h 03mDask: Next Steps in Parallel Python
01h 26mDask tutorial
01h 23mData Science at Scale with Dask
01h 56mParallelizing Scientific Python with Dask
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars
44mDask for ad hoc distributed computing