Talk Python To Me: Stream Processing in your favorite language with Beam on Flink
summary
Demonstrates a running design that lets Python Beam pipelines execute on Flink by cutting Python PTransforms out of the graph and running them in a separate gRPC SDK harness (optionally in Docker). Implements three portability APIs (pipeline, job, Fn/state/logging) and shows tradeoffs: Beam portability APIs enable cross-language connectors but add overhead — Google reports 10–15% on Dataflow while a local prototype shows up to 5x slowdown for remote-function RPCs; work is ongoing to finish integration.
AI-written summary · about
chapters (8)
- 0:00Introduction and setup
- 1:49What is Apache Beam
- 6:48Beam model and examples
- 17:24Portability APIs overview
- 23:41Pipeline and Job APIs details
- 27:25Fn API and execution flow
- 35:36Advantages and challenges
- 37:01Future work and Q&A
related talks
38mBeam on Flink: How does it actually work?
38mData processing with Apache Beam
53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]
40mIt’s finally here: Python on Flink & Flink on Zeppelin
35mPyFlink: Building Flink applications with Python language
25mBuilding a Python Data Pipeline with Apache Flink
42mApache Beam on Dask- Portable, Scalable, Scientific Python
39mHow Python Drives Modern Data Engineering