Moonshot Spark: serverless with GraalVM
summary
Argues that running Apache Spark serverless is feasible but constrained: GraalVM native images cut cold starts and memory use yet require workarounds for reflection and codegen; shuffle remains the hard problem and Google Cloud proves distributed NFS can work as an external shuffle; DataBricks already offers serverless autoscaling since 2017 and Palantir contributes Kubernetes shuffle/scheduler patches; proposes a long-running driver dispatches ephemeral executors model or standalone functions per task, notes limits on function runtime, I/O and resource quotas.
AI-written summary · about
chapters (9)
- 0:00Introduction to Moonshot Spark
- 3:37Serverless paradigm shift
- 7:49Stateless Spark functions
- 12:32Cloud serverless Spark options
- 16:33Shuffle and auto-scaling challenges
- 20:51Scheduler and pluggable architecture
- 23:07GraalVM and native images
- 26:20Python and developer experience
- 28:52Future of serverless Spark
related talks
36mServerless Java
35mServerless Experiences -- On Your Own Servers?
24mServerless Java hacks: Think different and leave old habits...
28mThe Brave New World of serverless computing
43mHow and why I turned my old Java project into a first-class serverless component
46mServerless Java: Challenges and Triumph
26mGoing Serverless With Spring’s Support for GraalVM, Project CraC & More
49mThe Java Cloud-Native Stack for Microservices and Serverless Architecture