Changing Data With Confidence using DuckDB
summary
DuckDB provides ACID all-or-nothing semantics so updates either fully commit or automatically roll back on error or crash; a demo shows a failed insert undone and a crash leaving only fully committed rows durable. It runs in-process with Python so in-process pandas zero-copy reads and Arrow interchange avoid socket overhead. Installer is ~15 MB pip package; it reads/writes Parquet/CSV, talks to S3/Postgres, and its MVCC concurrency lets concurrent dashboard reads and background writes proceed with low contention. Benchmarks show fast CSV load (≈1 GB test) and cheap updates even at 1–100% affected rows.
AI-written summary · about
chapters (21)
- 0:00Welcome to DuckDB
- 1:08DuckDB's mission
- 2:56DuckDB features
- 4:00Batteries-included design
- 5:26Performance and vectorized engine
- 6:53Licensing and funding
- 7:41Python integration
- 9:05Data change problems in Python
- 12:14ACID properties
- 13:38Atomicity in DuckDB
- 15:23Atomicity in analytics
- 17:11Consistency constraints
- 18:36Isolation and concurrency
- 19:39Durability and crash safety
- 21:49Performance myths
- 22:30Reading performance
- 23:35Update performance
- 25:03Summary
- 25:23Q&A on AI/ML workflows
- 27:14Q&A on testing
- 29:44Closing
related talks
27mState of the Duck
34mData Wrangling with DuckDB
40mDuckDB: Harnessing in-process analytics for data science and beyond
30mDuckDB: Bringing analytical SQL directly to your Python shell
28mAnalyze data easily with duckdb - and the implications on data architectures
25mProcessing Biggish Data with DuckDB and Python
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
27mIn Process Analytical Data Management with DuckDB