Challenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
Talkat KubeCon + CloudNativeCon Europe 2025 · on Thursday, 2025-04-03· at 17:30· Level 1 | Hall Entrance N10 | Room H·
summary
Case study of migrating Spark from bare metal YARN and Mesos to Kubernetes and then adding Unicorn scheduler to regain queuing, gang scheduling, priority preemption and fair sharing; uses shuffle tracking instead of an external shuffle service, uses cloud object storage (S3-like) for data, and separates workloads into three node pools (Jupyter, drivers, executors) to reduce shuffle latency and allow preemptable executors. Includes a JupyterLab kernel config manager, recommendations to use local disks for partition throughput, and reports measurable cost and utilization improvements.
AI-written summary · about
chapters (10)
- 0:00Spark migration journey
- 2:08Interactive Spark challenges
- 4:15Spark on bare metal limits
- 7:45Mesos to virtualized environments
- 9:53Kubernetes migration
- 13:29Jupyter kernel config
- 15:14Unicorn scheduler architecture
- 19:55Unicorn cost savings
- 21:44Spark on Kubernetes takeaways
- 26:27Q&A storage shuffling
related talks
30mBeyond Experimental: Spark on Kubernetes
26mRe-architecting the Data Platform with Spark on Kubernetes
33mKubernetizing Big Data and ML Workloads at Uber
28mCloud-Native Apache Spark: why & how to migrate your Spark
22mSPOK - Managing ML/Big Data Spark Workloads at scale on Kubernetes
29mScaling Apache Spark on Kube to Apple Scale
21mSpark on Kubernetes: The Elastic Story
31mSpark challenges