The Magic of Spark: How High-Level Pipelines Become Distributed Hardcore
summary
The talk explains how Apache Spark implements high-level data pipelines by generating and distributing low-level Java/bytecode (codegen) rather than simply running Java Stream operations remotely. It covers why operations like flatMap or groupBy cannot be naively invoked on remote machines and how Spark’s code generation makes distributed execution efficient.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 2:56What is Spark
- 10:47Code generation in Spark
- 21:18Spark API internals
- 35:51Codegen and optimization
- 45:02Generated code structure
- 51:18Spark app lifecycle
- 53:51Summary
related talks
01h 07mMyths about Spark, or Can an Ordinary Java Developer Use Spark?
53mLord of the Spark
34mBig Data, Small Code ... #bbuzz
01h 01mWhy should you wake up tomorrow and start to learn Scala?
47mStreams – Below the Surface
50mA Java Implementer's Guide to Better Apache Spark Performance
54mWhat Does the Future Spark Have in Store for Us?
49mML Pipelines