Speed is Not All You Need for Data Processing
summary
Benchmarks compare Pandas, Polars, DuckDB, Spark and Dask across sizes and show context matters: synthetic TPCH/H2O numbers report DuckDB 80x improvement but IO and query shape change rankings; a direct Pandas filter goes from 3s to 162ms by reordering, and Polars hits 15 ms in a tuned case; a 3 billion‑row job shows Spark handles 100 GB under one minute while DuckDB OOMs on modest CPU counts; an abstraction layer fug ports pandas code to Spark/Dask/DuckDB to avoid premature rewrites.
AI-written summary · about
chapters (12)
- 0:00Speed limits in data processing
- 1:51Speed benchmarks in data tools
- 3:38Context in benchmarking performance
- 6:53Manipulating benchmarks
- 8:19Big data is not dead
- 15:12Vendors and speed benchmarks
- 18:43No single tool wins
- 20:28Beyond speed in tool adoption
- 23:00Choosing tools for workloads
- 25:28Fug for multi-engine workflows
- 29:25Don’t trust benchmarks blindly
- 30:00Conclusion
related talks
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
35mHow many dataframe libraries do you need to change a lightbulb?
34mData Wrangling with DuckDB
25mProcessing Biggish Data with DuckDB and Python
30m"Better dataframes"
25mDistributed data workflows: PySpark vs Dask
18mLooking under the hood - demystifying data tools
28mAdapting from Spark to Dask: what to expect