Spark challenges
summary
ZipRecruiter migrates pipelines to Apache Spark running Spark on Kubernetes on AWS and repeatedly fixes failures by blindly increasing memory until joins and shuffles break. Practical fixes include detecting and splitting data skew (null-heavy keys or bot-driven outliers), preferring partition-aligned join columns, using AQE to coalesce post-shuffle partitions and convert sorts to broadcasts, avoiding extra actions by using accumulators, and sizing partitions for aim for 1–2 minute tasks. Examples and runnable snippets live on the presenter’s GitHub.
AI-written summary · about
chapters (19)
- 0:00Intro
- 2:01Topics
- 2:39Project stages
- 3:25Timeline
- 6:20Data distribution skew
- 7:38Input data skew at storage layer
- 8:27Data skew introduced during processing
- 8:42Symptoms of data skew
- 9:52Validate data skew (records per partition)
- 11:25Useful tools for stable data processing
- 12:46Properties of join operation
- 14:28Spark join strategies
- 15:50Adaptive query execution (AQE)
- 17:18Joins with skewed data - avoid them altogether?
- 18:09Redundant spark actions
- 19:44Fault tolerance vs parallelism
- 21:45Spark, JVM, memory
- 26:11Spark API is confusing when coming from Hadoop
- 28:00Spark terminology is confusing
related talks
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
26mRe-architecting the Data Platform with Spark on Kubernetes
34mKick-Ass Tools and Processes for Scalable Data Science
21mAvoiding Spark Pitfalls at Scale
29mDynamic Large Scale Spark on Kubernetes
30mBeyond Experimental: Spark on Kubernetes
24mTaming Data Skew in Production ML Pipelines
31mDynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events