Data infrastructure to build bigger with less
summary
Argues that most analytics no longer need clusters by showing concrete results: DuckDB runs in-browser via WebAssembly and on laptops, a 13 billion‑row group‑by + regex finishes in 53.6 seconds on a 2.5‑year‑old laptop vs ~21s on 1,100 machines and costs a fraction of cloud pricing; analysis of Redshift/Snowflake logs finds one in 500 queries truly requires distributed memory. Demonstrates hybrid/dual execution that plans joins across local DuckDB, MotherDuck cloud workers and S3 Iceberg storage, and reports it successfully reassembles results.
AI-written summary · about
chapters (8)
- 0:00Introduction to small data
- 1:18Key principles of the movement
- 3:12The history of big data
- 7:15Flaws in the big data story
- 10:47The power of single nodes
- 14:35Redefining big data scale
- 16:31Hybrid data architectures
- 19:30Q&A and concluding thoughts
related talks
51mMotherduck: Data Analytics in the Post Big-Data Era
27mDataEngBytes 2023 - MEL-T1-07 - Ryan Boyd
28mAnalyze data easily with duckdb - and the implications on data architectures
33mDataEngBytes 2023 - SYD-T1-01 - Ryan Boyd
45mAnalytics for not so big data with DuckDB
38mFrom Big Data to DuckDB: Ryan Boyd (MotherDuck) on Hyper-Tenancy, Duck Lake, and Making Data Fun.
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
30mSparkless Patterns: The New Data Stack and Modern Data Architecture