Faster Spark SQL: Adaptive Query Execution in Spark v3
summary
Introduces Apache Spark SQL's Adaptive Query Execution (AQE) in Spark 3, which uses runtime statistics to modify execution plans while queries run. Covers concrete features such as switching join strategies (broadcast-hash-join vs sort-merge-join), handling data skew, and dynamically choosing the number of shuffle/reducer partitions, and reports TPC-DS benchmark improvements.
AI-written summary · about
