Cloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available
summary
Spark on Kubernetes reaches Spark 3.1 GA with concrete features: dynamic allocation with shuffle tracking and graceful node decommissioning that moves shuffle files on spot preemption. Practical tooling includes cluster-autoscaler + multiple node pools, overprovisioning with “pause” pods for sub-10s startup, and published Docker images on Docker Hub. Example migration shows reduced AWS costs by two thirds, half the startup time and ~35% lower app duration; a separate case gets 10x speedup by adding local SSDs for shuffle.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:20About JeanYves
- 2:11Agenda
- 4:51Spark on Communities Operator
- 5:40Differences with Kubernetes
- 6:29Containerization
- 7:34Shared Infrastructure
- 8:12Free tooling
- 8:47Customer example
- 10:15Infrastructure management
- 12:10Over provisioning
- 13:36Dynamic allocation
- 15:12Docker images
- 16:38Monitoring
- 18:02Delight
- 20:05Customer Performance Improvement
- 21:24Spark on Kubernetes is generally available
- 22:51Better handling for node shutdown
- 25:08Shared NFS volume
- 25:56Demo
related talks
28mCloud-Native Apache Spark: why & how to migrate your Spark
40mApache Spark on Kubernetes: A Technical Deep Dive
21mSpark on Kubernetes: The Elastic Story
29mDynamic Large Scale Spark on Kubernetes
39mSpark on Kubernetes: Best Practice and Performance
31mManaging Multi-Cloud Apache Spark on Kubernetes
30mBeyond Experimental: Spark on Kubernetes
26mRe-architecting the Data Platform with Spark on Kubernetes