Avoiding Spark Pitfalls at Scale
summary
Production Spark on the Scala API focused on concrete fixes: data skew causes executors to OOM so split hot/cold keys and run a skew join by broadcasting the hot side and unioning results; increase broadcast timeout and tune spill thresholds to avoid memory deaths; prefer repartition over coalesce to avoid serializing huge upstream work; cache intermediate datasets to force evaluation and simplify planning; set parallelism ~2× cores (default 200); prefer select over map for column ops; consider frameless or Quilt for better catalyst optimization on large, multi‑billion row pipelines.
AI-written summary · about
chapters (15)
- 0:00Introduction and speaker intro
- 2:23Talk overview and scope
- 3:25Understanding data skew basics
- 5:54Impact of data skew in production
- 7:17Solving skew with broadcast joins
- 8:45Spark execution engine quirks
- 10:10Partitioning and coalescing pitfalls
- 11:58Lazy API and query planning issues
- 13:02Configuration and tuning tips
- 14:27Spill thresholds and memory tuning
- 15:10Cluster health and monitoring
- 16:11Spark API performance tricks
- 17:35Serialization and lazy evaluation
- 18:21Alternative Spark APIs and projects
- 19:30Summary and closing remarks
related talks
24mTaming Data Skew in Production ML Pipelines
36mNo more struggles with Apache Spark (PySpark) workloads in production
24mDistinguishing features of production quality pipelines
15mOptimizing Query Performance in Big Data Systems
41mSpeeding Up Data Pipeline Construction in Apache Spark
31mSpark challenges
43mOptimizing Spark applications from simple to complex with examples
42mApache Spark on planet scale