Best Practices and Optimization with the Spark Engine
summary
Overview gives concrete tactics not vagueness: Spark runs on Databricks, Microsoft Azure Synapse and Microsoft Fabric and processes 50 billion rows in examples; cluster sizing uses cores as width and RAM as height with 128 MB default partitions; Delta Lake tables use a transaction log and require optimize and vacuum to compact and remove tiny files; advanced fixes include deletion vectors for cheap updates and liquid clustering for file skipping; serverless clusters can auto‑scale to huge cost if bad queries run.
AI-written summary · about
chapters (26)
- 0:00Spark optimization in Delta Lake
- 1:45Spark engine overview
- 4:34Delta Lakehouse benefits
- 5:56Spark cluster & transformations
- 8:47Cluster config & serverless
- 9:27Memory partitioning & tasks
- 10:29Narrow transformations
- 12:36Wide transformations
- 13:39Catalyst Optimizer deep dive
- 15:46Predicate pushdown & folding
- 16:50Projection pruning & joins
- 17:31Language impact on performance
- 18:52DataFrame vs RDDs
- 19:33Delta tables structure
- 20:57Delta log & metadata
- 22:41Writing to Delta tables
- 23:44Delta versioning & time travel
- 24:46Medallion architecture
- 25:31Delta maintenance: optimize & vacuum
- 28:00Storage pitfalls & fixes
- 28:43Tiny files & over-partitioning
- 30:06Liquid clustering
- 32:33Deletion vectors
- 35:20Key takeaways
- 37:06Q&A: Spark & SQL Server
- 38:35Data quality auditing
related talks
41mOptimising Apache Spark and SQL for Improved Performance
17mHow changes to Azure Synapse Analytics impact data architecture designs
15mOptimizing Query Performance in Big Data Systems
50mChallenges that everyone struggles while productionizing Apache Spark workloads
28mAn Apache Spark query's journey through the stack
55mAzure SQL database, Azure Synapse Analytics, Azure Databricks?
28mAI-driven query plan optimization in big data processing tools
53mOptimizing your Azure SQL Database workloads