Less but Better: Toward Simpler and More Efficient Data Engineering…
summary
This talk compares traditional distributed big-data stacks with a simpler approach focused on single-node tools to reduce cost, complexity and data bloat. It examines the open-source projects DuckDB and Polars, explaining what workloads and problems they address for data engineers and data scientists and how far these mono-node tools can replace large distributed systems.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:09Tools overview
- 4:59Test environments
- 5:43First test 5 GB
- 6:48Second test 75 GB
- 8:17Third test 320 GB
- 9:20Cost analysis
- 10:03Summary
- 12:10Q&A
related talks
30mSparkless Patterns: The New Data Stack and Modern Data Architecture
24mData infrastructure to build bigger with less
51mMotherduck: Data Analytics in the Post Big-Data Era
41mPanel Discussion
28mAnalyze data easily with duckdb - and the implications on data architectures
38mFrom Big Data to DuckDB: Ryan Boyd (MotherDuck) on Hyper-Tenancy, Duck Lake, and Making Data Fun.
31mFuture of data engineering: what's in store?
02h 18mDistributed Systems: In the Daily Life of the Data Engineer