DuckDB: Bringing analytical SQL directly to your Python shell
summary
DuckDB is an embedded analytical database (pre-release v0.6) that executes SQL without a server and offers zero-copy integration with Pandas and Arrow, a Python DBI and a relational API. It uses columnar storage, compression (reported 3–5x compression on taxi benchmarks), vectorized execution with 2048-element batches, automatic parallelism and spill-to-disk for beyond-memory queries. Live demo on NYC taxi data shows 0.26s vs 1.37s for a group/aggregate and 0.7s vs 2.18s for a regression.
AI-written summary · about
chapters (6)
- 0:00Start
- 4:22Columnar Data Storage
- 6:21Compression
- 10:47End-To-End Query Optimization
- 12:17Automatic Parallelism & Beyond Memory Execution
- 21:26Summary
related talks
34mData Wrangling with DuckDB
27mIn Process Analytical Data Management with DuckDB
39mPython and SQL: Better Together, Powered by DuckDB
40mDuckDB: Harnessing in-process analytics for data science and beyond
01h 02mAnalytics for not-so-big data with DuckDB
25mA quack of all trades: DuckDB, a versatile analytical database to keep in your toolkit
28mAnalyze data easily with duckdb - and the implications on data architectures
25mProcessing Biggish Data with DuckDB and Python