Deep Dive: Spark Memory Management
summary
Covers how Spark actually divides JVM heap between execution and storage and the change to unified memory management (Spark 1.6) that lets execution borrow unused cache space to avoid unnecessary spills. Describes dynamic per-task sharing that grows with active tasks to mitigate stragglers and cooperative spilling between operators so a sorter can steal pages from an aggregator. Explains Tungsten optimizations: compact byte rows, cache-aware layouts and optional off‑heap allocation to reduce GC and I/O.
AI-written summary · about
chapters (9)
- 0:00Introduction to Spark
- 3:55Memory Management Challenges
- 6:49Execution vs Storage Memory
- 9:29Static Allocation Model
- 12:13Unified Memory Management
- 16:10Task-Level Arbitration
- 19:21Operator-Level Arbitration
- 23:56Project Tungsten Overview
- 28:32Q&A and Closing
related talks
55mSpark Architecture
50mA Java Implementer's Guide to Better Apache Spark Performance
21mAvoiding Spark Pitfalls at Scale
23mDemystifying Flink Memory Allocation and tuning
30mOptimize Spark Jobs Using Its UI
15mOptimizing Query Performance in Big Data Systems
46mSpark: One Tool for Everything?
53mLord of the Spark