In Process Analytical Data Management with DuckDB
summary
DuckDB is an in‑process analytical SQL engine that runs inside Python, R, JS or the browser (WASM) and reads Pandas/Arrow/Parquet in place, making data transfer near‑instant. Benchmarks show it scans 50 GB (1 billion rows) and runs five complex queries in 24 seconds on a workstation, roughly an order of magnitude faster than Spark and ~30× faster than Pandas. It supports multi‑core vectorized execution, joins of billion×billion rows, graceful out‑of‑RAM execution to SSD, and integrations with DBT, SQLAlchemy and MotherDuck cloud.
AI-written summary · about
chapters (13)
- 0:00Introduction to DuckDB
- 2:11The origin of DuckDB
- 4:09What is DuckDB?
- 6:07Comparing data tools
- 11:20Modernizing SQL
- 13:43Python integration
- 15:30Naming and mission
- 16:31Benchmarking performance
- 17:55Under the hood
- 21:13Scalability and deployment
- 22:59Ecosystem interoperability
- 24:25Tooling and library support
- 25:42Conclusion and summary
related talks
19mIn-Process Analytical Data Management with DuckDB
34mData Wrangling with DuckDB
28mAnalyze data easily with duckdb - and the implications on data architectures
40mDuckDB: Harnessing in-process analytics for data science and beyond
30mDuckDB: Bringing analytical SQL directly to your Python shell
01h 02mAnalytics for not-so-big data with DuckDB
39mPython and SQL: Better Together, Powered by DuckDB
40mDuckDB beyond the notebook