Interactive Spark at Your Fingertips: Integrating SparkConnect into Kubeflow Noteboo...
summary
Shows how to run real interactive Spark inside Kubeflow Notebooks by deploying a SparkConnect server via the Spark Operator and controlling sessions from the Kubeflow SDK. Demo spins up a driver plus 3 executors, lets Python and Scala clients (PySpark and Almond) connect, and supports per-session driver/executor sizing and delete-session cleanup. Notes operational caveats: Istio sidecars must be bypassed, correct service account and typing-extensions >4.13 needed, and SparkConnect has no built-in TTL.
AI-written summary · about
chapters (14)
- 0:00Interactive Spark on Kubeflow
- 1:44Why interactive Spark matters
- 3:51Spark integration challenges
- 4:14Spark Connect introduction
- 6:44Spark Connect architecture
- 7:26Spark Connect deployment
- 9:10Demo setup prerequisites
- 13:44Running Spark Connect demo
- 15:29Scala notebooks to Spark Connect
- 17:39Troubleshooting common issues
- 19:27Scala support in Spark Connect
- 20:08Q&A: Kubeflow notebooks
- 21:59Q&A: Deployment methods
- 24:13Closing remarks
related talks
33mSpark Operator - Feature Engineering with Spark on Kubeflow
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
21mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
29mDynamic Large Scale Spark on Kubernetes
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
55mUse Spark from Anywhere
45mCan you build a cloud‑native application with Spark?
40mApache Spark on Kubernetes: A Technical Deep Dive