Scaling Outside the Warehouse Using DuckDB and Python
summary
DuckDB integrates with Python to run in‑RAM OLAP SQL on local files, S3 and https, and scales via AWS Lambda or Coiled EC2 workers. The demo runs 30M rows (nine months) transforming in ~0.27s and converting to pandas in ~26s, and scales to 60 parallel tasks with latency staying ~13s. Cost examples show 92–96% cost reduction versus Snowflake, and setup uses DuckDB+Python+Coiled+Lambda while observing Lambda 10 GB RAM 15 min max.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:52DuckDB game-changer for data
- 5:27DuckDB basics and Python
- 9:45Python and SQL workflows
- 15:4730M rows in under a second
- 16:08Scaling with AWS Lambda
- 21:5560 months of data in 13s
- 23:45Coiled cloud computing
- 28:10Live demo: Coiled notebook
- 29:15Q&A
related talks
25mProcessing Biggish Data with DuckDB and Python
34mData Wrangling with DuckDB
28mAnalyze data easily with duckdb - and the implications on data architectures
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB
27mIn Process Analytical Data Management with DuckDB
40mDuckDB: Harnessing in-process analytics for data science and beyond
40mDuckDB beyond the notebook
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers