Data Wrangling [for Python or R] Like a Boss With DuckDB
summary
DuckDB runs in process with zero dependencies and parses messy files faster than a server workflow so an 857 MB New Zealand census CSV with 34 million rows becomes queryable and a complex multifile aggregation completes in about 1.1 seconds on a laptop. It uses a parallel vectorized engine, a world class CSV reader, native Parquet and Arrow support so CSV -> Parquet: 71 MB shrinks storage, and it stores tables views and macros in one file with atomic updates and transactional rollbacks.
AI-written summary · about
chapters (8)
- 0:00DuckDB and data wrangling intro
- 10:08Frustrations with traditional databases
- 18:59DuckDB’s in-process analytics
- 26:16DuckDB meets dplyr
- 30:30Atomic datasets and advanced features
- 37:18Future of single-node processing
- 46:32DuckDB extensions and growth
- 48:18Q&A
related talks
31mData wrangling like a boss with Duckdb
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
34mData Wrangling with DuckDB
40mDuckDB: Harnessing in-process analytics for data science and beyond
27mIn Process Analytical Data Management with DuckDB
01h 02mAnalytics for not-so-big data with DuckDB
25mProcessing Biggish Data with DuckDB and Python
19mIn-Process Analytical Data Management with DuckDB