PyFlink: Building Flink applications with Python language
summary
PyFlink adds Python Table API/SQL and DataStream API support enabling Python UDF, UDTF, UDAF, Pandas UDF/UDAF and uses Py4J to map Python APIs to JVM jobs. Runtime builds Java operators plus Python workers, optimizes with operator fusion, batches Arrow data for smaller serialization and faster RPCs, and supplies custom type serializers and multi-level async state reads. Demos show a streaming wordcount and per-user order aggregation; checkpoints are currently pipelined and hard to predict, and a prototype of in-JVM UDF execution shows promising speedups.
AI-written summary · about
chapters (4)
related talks
40mIt’s finally here: Python on Flink & Flink on Zeppelin
25mBuilding a Python Data Pipeline with Apache Flink
41mTalk Python To Me: Stream Processing in your favorite language with Beam on Flink
19mFlow-based programming made easy with PyF 2.0
38mBeam on Flink: How does it actually work?
31mBringing the Jewels of the Python World to Scala with Spark
25mBuilding an End-to-End Analytics Pipeline with PyFlink
38mData processing with Apache Beam