Optimizing Spark applications from simple to complex with examples
summary
Covers practical Spark optimization techniques: using a custom script plus manual and automated repartitioning to improve data processing, tuning window functions for performance, and different join strategies including cases where avoiding joins is preferable.
AI-written summary · about
chapters (8)
- 0:00Spark optimization intro
- 2:29Spark repartitioning
- 9:53Optimizing joins
- 15:57Window functions
- 20:30Parallel execution
- 27:18Managing Spark apps
- 33:03Automating config recalc
- 34:09Q&A
related talks
30mOptimize Spark Jobs Using Its UI
15mOptimizing Query Performance in Big Data Systems
21mAvoiding Spark Pitfalls at Scale
36mNo more struggles with Apache Spark (PySpark) workloads in production
46mSpark: One Tool for Everything?
50mIntroduction to Real-Time Big Data with Apache Spark
52mSolving Analytical Problems using Apache Spark
41mSpeeding Up Data Pipeline Construction in Apache Spark