Composable Parallel Processing in Apache Spark and Weld
summary
Explains how Apache Spark’s DataFrames and Datasets expose a schema-aware DSL and runtime (Spark SQL/Tungsten) that enables filter pushdown, runtime codegen and resulted benchmarks showing up to 4x throughput on a Yahoo streaming workload and hundreds of production streaming apps (largest process tens of trillions of records/month). Proposes Weld: a Rust/LLVM-backed lazy IR with Weld IR and builders that fuses across libraries and hardware to deliver ~5–10x per-library speedups and up to ~30x when optimizing cross-library.
AI-written summary · about
chapters (9)
- 0:00Introduction to talk
- 2:26Challenges in parallel computing
- 5:33Composition in Apache Spark
- 10:04Structured APIs in Spark
- 18:36Benefits of structured APIs
- 27:09Weld research project
- 31:02Weld’s API and IR design
- 37:03Weld performance results
- 38:52Q&A session
related talks
31mWeld: An Optimizing Runtime for High Performance Data
36mIn-memory data pipeline and warehouse at scale
55mSpark Architecture
39mComplex Data Transformations Made Easy
41mSpeeding Up Data Pipeline Construction in Apache Spark
30mSparkless Patterns: The New Data Stack and Modern Data Architecture
28mDistributed Computation with WASM and WASI
11mScaling R with Spark