Dynamic Large Scale Spark on Kubernetes
summary
Shows how to run Spark on Kubernetes with concrete choices: use the Spark Operator, keep the driver on on‑demand instances and executors on spot instances for cost savings, and use Carpenter to auto‑provision nodes for pending pods (demo proves nodes appear and disappear on job start/finish). Uses Argo Events to ingest SQS messages and trigger Argo Workflows, demonstrates multi‑version/multi‑arch images (Spark 3.3 vs 3.4, x86 and Graviton), and warns about VPC IP exhaustion and secondary CIDR sizing.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:26Goals of scaling Spark on K8s
- 2:31Apache Spark and K8s integration
- 3:56Benefits of Spark in containers
- 6:06Key scaling considerations
- 8:18Networking and IP in cloud K8s
- 9:42Compute options for Spark
- 11:09Autoscaling Spark workloads
- 12:59K8s scheduler limits and alternatives
- 14:26Argo Workflows and Events
- 15:51Event-driven with Argo Events
- 16:57Demo setup and repo
- 19:35Live demo execution
- 26:26Post-demo cleanup
- 27:31Q&A
related talks
31mDynamic Large Scale Spark on Kubernetes: Empowering the Community with Argo Workflows and Argo Events
21mSpark on Kubernetes: The Elastic Story
22mSPOK - Managing ML/Big Data Spark Workloads at scale on Kubernetes
40mApache Spark on Kubernetes: A Technical Deep Dive
14mLightning Talk: Running Cloud-native Spark Jobs with Argo Workflows
36mAutomating Cloud-native Spark Jobs with Argo Workflows
30mBeyond Experimental: Spark on Kubernetes
39mSpark on Kubernetes: Best Practice and Performance