Storage-Partitioned Join for Apache Spark
summary
Proposal adds storage-partitioned joins to Spark v2 data sources, aiming to skip expensive network shuffles by matching disk partitions instead of always hashing both sides. Implements partition grouping and correlize/repartition strategies to handle mismatched bucket counts, unordered or subset/superset keys, and heavy skew via partition splitting. Prototype supports inner joins and custom bucket/hash functions for Hive and Iceberg, exposes a new partially clustered distribution to increase parallelism, and targets inclusion in Spark 3.3 with a PoC available now.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 4:28Problems with Bucketed Joins
- 5:55Bucketed Joins in Spark
- 7:02The Problem
- 10:06Transforms
- 11:28Storage Partition Join
- 12:42Examples
- 14:15Cases where partitions do not match
- 22:49Grouping partitions with correlated partitions
- 26:10Grouping partitions with skewed partitions
- 30:14Partitioning with proper size
- 31:54Questions
related talks
41mOptimising Apache Spark and SQL for Improved Performance
50mChallenges that everyone struggles while productionizing Apache Spark workloads
42mApache Spark on planet scale
28mAn Apache Spark query's journey through the stack
15mOptimizing Query Performance in Big Data Systems
31mSpark challenges
43mBeyond Shuffling: Scaling Apache Spark
35mDataFrames in Apache Spark - the analysts perspective