KEYNOTE: Data Architecture Turned Upside Down
summary
Claim: flip the stack so compute moves to clients while storage and metadata stay commoditised. Concrete evidence: count distinct on a billion rows in 46s on a laptop with 2 GB memory, demos on phones and Raspberry Pi, and benchmarks showing most queries scan ~100 MB. Tools and formats: single-node beyond-memory intermediates, DuckDB + DuckLake with Parquet on object store and a lightweight catalog. Outcome: local query processing scales naturally with users and cuts cloud warehouse cost.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 3:39What is DuckDB
- 6:301985 data architecture model
- 8:17Costs of traditional warehouses
- 10:44Thin straw in data transfer
- 12:11Python and pandas revolution
- 13:40Need for Spark?
- 15:52Query sizes and scaling
- 16:36DuckDB 265GB demo
- 19:47DuckDB beyond memory
- 20:30Complex tasks on old hardware
- 23:21Chaos in file-based data
- 24:44Lakehouse limitations
- 27:16Iceberg reinvented DuckLake
- 29:48Flipping data architecture
- 33:21Local data processing
- 34:25DuckDB on edge devices
- 36:55Summary Q&A
related talks
51mMotherduck: Data Analytics in the Post Big-Data Era
30mRetooling for a Smaller Data Era
27mDataEngBytes 2023 - MEL-T1-07 - Ryan Boyd
30mSparkless Patterns: The New Data Stack and Modern Data Architecture
58mUnderstanding Big Data for Software Engineers
11mBreaking Data Silos: Enterprise Data Management
30mDataEngBytes 2023 - BRI-T1-08 - Ryan Boyd
34mData Storage: NoSQL Toasters and a Cloud of Kitchen Sinks