DuckDB: Harnessing in-process analytics for data science and beyond
summary
DuckDB is an in-process columnar SQL engine that runs inside applications and even WebAssembly in browser, exposing zero‑copy access to Arrow, Pandas and Polars and reading Parquet/CSV/JSON over HTTP(S). The demo loads 423 million rows in 26s (CSV sniffer infers schema), returns an aggregation over 185M rows in <60ms — 3 billion rows/sec extrapolated — and does joins in a few hundred ms. Vectorized 248‑row vectors, zone‑map pruning, extensions (httpfs, parquet) and graceful larger‑than‑memory execution all actually work.
AI-written summary · about
chapters (12)
- 0:00Introduction to DuckDB
- 2:00Architecture and usage
- 6:29Installation and portability
- 9:00Analytical performance demo
- 19:05Internals and optimization
- 23:55Indexing and zone maps
- 25:59Graceful memory degradation
- 27:19SQL extensions and features
- 30:07Extensibility and ecosystems
- 32:06Beyond analytics
- 36:13Limitations and scaling
- 37:22Roadmap and conclusion
related talks
34mData Wrangling with DuckDB
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
27mIn Process Analytical Data Management with DuckDB
28mAnalyze data easily with duckdb - and the implications on data architectures
01h 02mAnalytics for not-so-big data with DuckDB
19mIn-Process Analytical Data Management with DuckDB
40mDuckDB beyond the notebook
30mDuckDB: Bringing analytical SQL directly to your Python shell