Sparkless Patterns: The New Data Stack and Modern Data Architecture
summary
Claims that DuckDB Polars and Apache DataFusion let small nodes replace heavy Spark clusters with concrete results: a Lambda running DuckDB reads Iceberg on S3 to serve SQL queries but hits cold‑start and memory‑leak issues, moving to a Kubernetes service fixes cache churn; reimplementing the query in Apache DataFusion (Rust) reads 10s of GB (hundreds of millions of rows) on a 4‑core pod with the lowest latency; replacing a Spark ETL with Polars halves runtime and cuts memory by ~75% using S3+SQS+Lambda for elastic scaling. DuckDB in‑process cache, DataFusion single‑node latency.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 3:18What are Sparkless Patterns
- 7:57Underlying technologies
- 11:28DuckDB, Polars, Data Fusion overview
- 17:11Case 1: API use case
- 24:45Case 2: Modern ETL
- 28:40Summary and recommendations
related talks
24mData infrastructure to build bigger with less
28mAnalyze data easily with duckdb - and the implications on data architectures
37mSpark, Dask, DuckDB, and Polars: Benchmarks
06mA quack at building scalable data pipelines with DuckDB
40mDuckDB beyond the notebook
51mMotherduck: Data Analytics in the Post Big-Data Era
40mDuckDB: Harnessing in-process analytics for data science and beyond
26mBuilding Data Lake Using DuckDB