Problem Solving Recipes Learned from Supporting Spark
summary
Covers concrete fixes: JVM OOM occurs on driver vs executor memory regions with spark.memory.fraction 60% default and storage fraction protections; prefer DataFrame/Dataset (Tungsten) to cut object footprint and run spark-lint before tuning. Dependency collisions cause NoSuchMethodError — upgrade, enforce classpath or shade Commons Math. Aim for two to three times cores partitions, tune speculation (raise speculation quantile 75%→95%), use broadcast joins but keep payload under ~2GB, handle S3 streaming by pacing or custom S3n handler.
AI-written summary · about
chapters (14)
- 0:00Start
- 2:52Prologue
- 3:27Memory Problems?
- 3:43Out Of Memory (OOM)
- 8:08OOM Tips
- 17:07Perplexities of Size
- 20:10Struggles in Speculation
- 23:52Common Issues
- 24:02Slow Joins
- 25:03Join Strategies
- 25:48Notes in the Margin: Broadcast Tips
- 26:41Unavoidable Joins
- 27:37Safe Stream Recovery
- 33:41Solutions
related talks
43mBeyond Shuffling: Scaling Apache Spark
38mPySpark in Practice
36mApplied Software Engineering Case Diary: Field Lessons in Problem Solving
59mSpark 2.0
39mSparklens: understanding the scalability limits of Spark applications
53mTips and tricks for robust big data applications
44mProcessing large data with Apache Spark
03mDBTB INT Andrew Or r