Cloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available

on YouTube ↗01m

" summary

Spark on Kubernetes reaches GA with Spark 3.1 (March) and is declared production ready; demonstrates running driver and executors as Kubernetes pods, native Kubernetes scheduler integration, and dynamic allocation with pod templates. Live code deploys a Spark application to a cluster, measures startup overhead and executor scaling, and compares cost and latency against Databricks and EMR and an on‑prem setup. Covers concrete config knobs, observed failure modes, and which workloads benefit from containerized isolation.

AI-written summary · about

" related talks

• 28mCloud-Native Apache Spark: why & how to migrate your SparkJean-Ives Stephan · Scale By the Bay 2020
• 40mApache Spark on Kubernetes: A Technical Deep DiveYinan Li · KubeCon + CloudNativeCon China 2018
• 21mSpark on Kubernetes: The Elastic StoryBowen Li, Huichao Zhao · KubeCon + CloudNativeCon Europe 2022
• 29mDynamic Large Scale Spark on KubernetesOvidiu Valeanu · KCD Romania 2024
• 39mSpark on Kubernetes: Best Practice and PerformanceJunjie Chen, Jerry Shao · KubeCon + CloudNativeCon China 2018
• 26mBuilding Apache Spark Clusters on K8s for AI WorkloadsAnant Kumar · Conf42 Cloud Native 2025
• 31mManaging Multi-Cloud Apache Spark on KubernetesIlan Filonenko, Aki Sukegawa · Kubernetes AI Day Europe 2022
• 26mRe-architecting the Data Platform with Spark on KubernetesSeungYong Oh · Kubernetes Forum Seoul 2019