Cloud-Native Apache Spark: why & how to migrate your Spark
summary
Spark runs natively on Kubernetes since 2.3 and is declared production ready in Spark 3.1; practical migration uses the spark-on-kubernetes operator, cluster autoscaler and labeled node pools so drivers run on on‑demand nodes while executors use spot instances. Key configs: set executor CPU requests lower than spark.executor.cores to avoid pending pods, enable dynamic allocation via shuffle tracking (Spark 3.0+) and mount local SSDs 10x IO (hostPath) to speed shuffle; use a Spark history server or remote event logs and consider pause pods overprovisioning to cut startup latency.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:55Spark basics and Kubernetes fit
- 7:48Why migrate to Kubernetes
- 12:03Migration checklist and setup
- 13:28Performance best practices
- 17:39Cost optimization strategies
- 20:54Monitoring Spark on Kubernetes
- 24:50Future of Spark on Kubernetes
- 27:56Q&A
related talks
29mCloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available
01mCloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available
40mApache Spark on Kubernetes: A Technical Deep Dive
29mDynamic Large Scale Spark on Kubernetes
21mSpark on Kubernetes: The Elastic Story
39mSpark on Kubernetes: Best Practice and Performance
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
26mRe-architecting the Data Platform with Spark on Kubernetes