Analytics for not so big data with DuckDB
summary
Claims that an in-process library analytical DB (DDB/DuckDB) handles the 1–100 GB “medium” workload and runs as a single binary or embeddable Python package. Demonstrations load 6 million rows and a 44 GB dataset with 1 billion rows, run a pivot aggregation in 8s and a billion-by-100M join in ~32s on a laptop, convert CSV→Parquet for ETL, query Parquet on S3 with downloaded 6KB via metadata/predicate pushdown, and integrate with cloud storage and notebooks successfully.
AI-written summary · about
chapters (14)
- 0:00🚀 Introduction: Working with Medium-Sized Data
- 3:00Understanding Big Data vs Medium Data
- 6:00Dark Data: The Challenge of Unused Data
- 8:00Tools for Medium-Sized Data: Why DDB is Ideal
- 12:00What is DDB? An In-Memory Analytical Database
- 15:00TCP-H Data Set: A Practical Example
- 18:00Running Queries: Performance with 6 Million Records
- 22:00Scaling Up: Querying a 1 Billion Record Database
- 26:00The Power of DDB: Pivoting Data & Aggregations
- 30:00Efficient Analytical Queries on Medium-Sized Data
- 34:00Using DDB in Python and Embedding Analytical Queries
- 38:00Transforming Data on the Fly: Using DDB for ETL
- 42:00Integrating DDB with Cloud Storage for Seamless Data Processing
- 44:00Conclusion: Why DDB is a Game-Changer for Medium Data
related talks
01h 02mAnalytics for not-so-big data with DuckDB
25mProcessing Biggish Data with DuckDB and Python
34mData Wrangling with DuckDB
28mAnalyze data easily with duckdb - and the implications on data architectures
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
33mData preparation and analytics on your laptop with DuckDB
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB
40mDuckDB: Harnessing in-process analytics for data science and beyond