Use Spark from Anywhere

on YouTube ↗55m

" summary

Spark Connect (shipped in Spark 3.4) decouples client and server so applications use the DataFrame/Dataset API (no RDD support) by sending unresolved logical plans over protobuf + Arrow. A tiny thin client runs in IDEs, notebooks, servers or phones, streams Arrow results, and lets IDEs debug client code while the cluster executes jobs. UDFs must be preinstalled on the server; databricks connect V2 is built on Spark Connect and the protocol enables Go, Rust and Scala clients.

AI-written summary · about

" chapters (9)

1. 0:00Spark from Anywhere intro
2. 1:25Spark evolution and challenges
3. 7:16Spark Connect architecture
4. 13:22Multi-tenant operational issues
5. 20:44Spark Connect demo setup
6. 35:18Connect to Databricks cluster
7. 41:57Debugging and IDE integration
8. 43:41Go and Rust clients
9. 54:01Q&A

" related talks

• 19mWhat's Next for Apache Spark™ Including the Upcoming Release of Apache Spark 4.0Reynold Xin, Tareef Kawaf · Data + AI Summit 2024
• 11mThe Evolution of Apache Spark™Reynold Xin · Data + AI Summit 2024
• 33mTo Spark or Not to SparkRussell Alexander Spitzer · Scale By The Bay 2019
• 24mInteractive Spark at Your Fingertips: Integrating SparkConnect into Kubeflow Noteboo...Vikas Saxena · Kubeflow Community Showcase 2026: GenAI and MLOps in Action
• 02mPreview of Apache Spark 4.0 and GA of Spark ConnectReynold Xin · Data + AI Summit 2024
• 47mKeeping the “fun” in Apache Spark Datasets and FPHolden Karau · Scala Days New York 2018
• 31mUsing Spark to accelerate your lakehouse architecture with FabricJustyna Lucznik · Microsoft Build 2023
• 28mBridging Python and Lakehouse worlds with Arrow Flight and DuckDB.Jim Dowling · Scale By the Bay 2023