Analytics for not-so-big data with DuckDB
summary
DuckDB runs analytical SQL on a single machine, demonstrating 1 billion rows in 15s on a laptop by fully using CPU, memory and vectorized ops. Live demos show reading Parquet over HTTP by fetching tiny headers, joining remote Parquet with a local MySQL table, and using it in-process: embed DuckDB in Python to query pandas DataFrames and return results as DataFrames. Quick ETL examples convert large CSVs into compact Parquet files with convert CSV to Parquet with one command.
AI-written summary · about
chapters (14)
- 0:00Medium data analytics
- 8:04What is DuckDB
- 13:24DuckDB demo with TPCH
- 21:46Querying billion-row data
- 24:36Optimizing analytical queries
- 29:39Key features for speed
- 31:23DuckDB in Python
- 33:50DuckDB with pandas
- 37:47Reading/writing file formats
- 43:10Parquet storage formats
- 48:12Querying remote Parquet
- 51:18Joining remote MySQL
- 57:47Simple ETL with DuckDB
- 1:01:43When to use DuckDB
related talks
45mAnalytics for not so big data with DuckDB
28mAnalyze data easily with duckdb - and the implications on data architectures
34mData Wrangling with DuckDB
40mDuckDB: Harnessing in-process analytics for data science and beyond
19mIn-Process Analytical Data Management with DuckDB
27mIn Process Analytical Data Management with DuckDB
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
25mA quack of all trades: DuckDB, a versatile analytical database to keep in your toolkit