Processing Biggish Data with DuckDB and Python
summary
Claims that DuckDB lets Python replace Spark/Trino for "biggish" workloads by running an in-process SQL RDBMS that avoids pandas memory limits and pushes data into L1 cache. Demonstrations load a 4 GB airline CSV (≈72 million rows) interactively, compose transformations with a composable Python API instead of stringy SQL, and run a 55 GB NYC taxi aggregate on a laptop in about 40 seconds (sometimes minutes). Integrates with Airflow/dbt and succeeds as a lightweight ETL engine.
AI-written summary · about
chapters (8)
- 0:00Introduction and speaker intro
- 3:58What is big data really
- 5:25Introducing DuckDB for bigish data
- 8:42ETL challenges and DuckDB benefits
- 12:22Python and DuckDB integration
- 17:12DuckDB in Python demo
- 23:07Handling 55GB dataset demo
- 24:12Summary and closing remarks
related talks
45mAnalytics for not so big data with DuckDB
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
27mIn Process Analytical Data Management with DuckDB
34mData Wrangling with DuckDB
01h 02mAnalytics for not-so-big data with DuckDB
30mDuckDB: Bringing analytical SQL directly to your Python shell
39mPython and SQL: Better Together, Powered by DuckDB
58mData Wrangling [for Python or R] Like a Boss With DuckDB