Can you build a cloud‑native application with Spark?
summary
A demo builds a container image containing Spark and an API that returns each speaker’s slot list plus feedback count and average rating, then deploys it on Kubernetes using Google’s Spark Operator CRD (Spark on Kubernetes). Logs stream to stdout, results are written to S3, and the demo shows drivers and executors starting; practical problems include Hadoop’s old AWS SDK preventing IAM role usage (forcing an IAM user), poor S3 small-file performance, and significant operational overhead, so a managed Spark service is still recommended as the first option.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:48Spark classic app
- 6:19Cloud-native app definition
- 9:2912-factor comparison
- 15:23Cloud-native refactoring
- 18:33Spark Operator deployment
- 24:33Live demo
- 30:54AWS compatibility issues
- 34:29Summary and recommendations
- 39:49Q&A
related talks
29mDynamic Large Scale Spark on Kubernetes
01mCloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available
21mSpark on Kubernetes: The Elastic Story
29mCloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available
36mAutomating Cloud-native Spark Jobs with Argo Workflows
30mBeyond Experimental: Spark on Kubernetes
28mCloud-Native Apache Spark: why & how to migrate your Spark
05mInterview