"Serpents and Ducks: wrangling data with Python and DuckDB"
summary
Demonstration shows DuckDB Python client ingesting varied file formats, integrating with Pandas/Ibis/Polars, and using the spatial extension to build a persisted cafe-finder from Melbourne’s open API. It processes a 5.6M-row CSV for sampling in ~2s and an aggregated month-by-month query in 0.1s for aggregation, exposes a lazy relationship object for delayed execution, reads 72,000 JSON files in one liner, performs S3 range scans and writes Parquet/CSV outputs, and all examples complete successfully.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:27What is DuckDB
- 6:51DuckDB vs pandas
- 12:12Exporting data & Python
- 17:05DuckDB data formats
- 22:17Finding a Melbourne café
- 26:36Summary
related talks
34mData Wrangling with DuckDB
25mA quack of all trades: DuckDB, a versatile analytical database to keep in your toolkit
25mProcessing Biggish Data with DuckDB and Python
27mIn Process Analytical Data Management with DuckDB
01h 02mAnalytics for not-so-big data with DuckDB
39mPython and SQL: Better Together, Powered by DuckDB
33mData preparation and analytics on your laptop with DuckDB
30mDuckDB: Bringing analytical SQL directly to your Python shell