Beam on Flink: How does it actually work?
summary
Explains how the Flink Runner translates Beam pipelines: the portable runner sends a protobuf Runner API to a job server which stages artifacts and launches language-specific SDK harness processes (Docker or shell) to execute Python/Go/Java code. Composite transforms become executable stages inside Flink that forward elements, state requests and side inputs to the SDK; timers are implemented as timers as elements emitted and consumed so they fire without blocking. Claims full-featured integration, cross-language pipelines work, and the runner is exercised by ValidatesRunner tests.
AI-written summary · about
chapters (6)
- 0:00Introduction and setup
- 1:51What is Apache Beam
- 8:00What is Apache Flink
- 13:20Classic Flink Runner
- 25:13Portable Flink Runner
- 36:47Conclusion and resources
related talks
41mTalk Python To Me: Stream Processing in your favorite language with Beam on Flink
38mData processing with Apache Beam
40mIt’s finally here: Python on Flink & Flink on Zeppelin
42mApache Beam on Dask- Portable, Scalable, Scientific Python
35mPyFlink: Building Flink applications with Python language
25mBuilding a Python Data Pipeline with Apache Flink
53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]
45mWhy Spring Belongs In Your Data Stream (From Edge to Multi Cloud)