Spark Execution Plans for Databricks
summary
Shows how Databricks uses the Catalyst Optimizer and explain() to turn code into logical then physical plans, with examples: a range 1..1,000,000 DF, repartition to 6 then 8, a join that triggers broadcast hash join vs sort‑merge, and Delta tables sized 30 million sales / 1 billion items. Explains explain modes (extend, codegen, cost, format), points out about four exchanges (shuffles), default 200 shuffle partitions, and how adaptive query execution and query hints change plans.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:32Understanding Spark plans
- 1:58Catalyst optimizer flow
- 6:38Reading execution plans
- 14:05Delta table plan analysis
- 25:51Using the Spark UI
- 29:27Adaptive query execution
- 36:22Advanced tips and recap
related talks
15mOptimizing Query Performance in Big Data Systems
43mOptimizing Spark applications from simple to complex with examples
45mSimplify your ETL with Databricks Delta Lake
21mAvoiding Spark Pitfalls at Scale
55mSpark Architecture
30mOptimize Spark Jobs Using Its UI
40mSpark Analytics on Cassandra Data
57mScaling Databricks to Run Data & AI Workloads on Millions of VMs