DataEngBytes 2023 - SYD-T1-01 - Ryan Boyd
summary
Argues that true “big data” is rare: 95% of BigQuery customers have <1TB and 90% of queries touch <100MB, so distributed clusters are often overkill. Demonstrates DuckDB running queries on 1.5 billion taxi rows in about 0.145s on a single machine. Describes MotherDuck’s prototype that embeds DuckDB in client and server (including WebAssembly) so the planner decides whether to run parts locally or in cloud, reducing distributed complexity while keeping performance.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:07Web apps to database engineering
- 3:55Myth of ever-growing big data
- 6:24Big data is dead long live easy data
- 10:01Hardware advances in data analytics
- 13:28Scale up versus scale out
- 17:24DuckDB and MotherDuck origins
- 21:38Local and cloud compute power
- 24:46Industry trends in data analytics
- 26:35Q&A
related talks
30mDataEngBytes 2023 - BRI-T1-08 - Ryan Boyd
38mFrom Big Data to DuckDB: Ryan Boyd (MotherDuck) on Hyper-Tenancy, Duck Lake, and Making Data Fun.
51mMotherduck: Data Analytics in the Post Big-Data Era
24mData infrastructure to build bigger with less
27mDataEngBytes 2023 - MEL-T1-07 - Ryan Boyd
45mAnalytics for not so big data with DuckDB
01h 02mAnalytics for not-so-big data with DuckDB
25mProcessing Biggish Data with DuckDB and Python