Life Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
summary
DuckDB, Daft, Polars and DataFusion are compared by concrete outcomes: a workload shows 2× performance boost in under a year, a 150 GB CSV processes on a single node with 2 cores and 16 GB RAM, and open table formats let local Python engines read cloud-style tables. Cache behavior to remote storage becomes the real bottleneck, some engines stream and spill to disk while others need more cores, and SQL interoperability often wins over proprietary Python APIs.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:06Data storage and compute separation
- 3:15Open table formats
- 7:14Modern data engines compared
- 10:45Pandas dominance reasons
- 11:50Apache Arrow exchange
- 12:10TPC-H performance tests
- 16:55Small vs big data
- 18:45150 GB CSV tests
- 21:17No standard API
- 22:19Query interface standardization
- 23:45Real-world minimal hardware
- 25:10Conclusion
related talks
34mData Wrangling with DuckDB
30mSpeed is Not All You Need for Data Processing
25mProcessing Biggish Data with DuckDB and Python
39mPython and SQL: Better Together, Powered by DuckDB
35mHow many dataframe libraries do you need to change a lightbulb?
30m"Better dataframes"
29m"Serpents and Ducks: wrangling data with Python and DuckDB"
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars