Building a Python Data Pipeline with Apache Flink
summary
Combines Python and Apache Flink to run scalable, stateful stream pipelines using the PyFlink Table API and Pandas vectorized UDFs to cut serialization overhead. Example pipeline reads Postgres via JDBC, applies a pretrained LDA model and tokenizer inside a Pandas UDF, groups and aggregates topics, and runs on a local Docker Compose Flink cluster with the web UI on port 8081. Requires Python 3.5+, successfully produces monthly topic counts and charts for a very large mailing-list archive.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:50Welcome
- 1:13Who am I
- 1:54Why combine python and Apache Flink
- 2:59Outline
- 3:27How does this affect you
- 4:21What is Flink
- 6:22Flink APIs
- 9:38Python APIs
- 12:06UDF Support
- 13:58Pandas Integration
- 15:31Vectorized Functions
- 17:34Example
- 19:02Creating Tables
- 19:48Writing a UDF
- 21:20Speaker Notes
- 21:39Running the Job
- 22:37Visualizations
- 22:55Recap
- 23:33Thank you
related talks
25mBuilding an End-to-End Analytics Pipeline with PyFlink
35mPyFlink: Building Flink applications with Python language
40mIt’s finally here: Python on Flink & Flink on Zeppelin
41mTalk Python To Me: Stream Processing in your favorite language with Beam on Flink
39mCodeless Pipelines With Pulsar and Flink
43mA Data Streaming Architecture with Apache Flink #bbuzz
39mStreaming doesn’t have to be hard
35mBringing Your Data Pipeline into The Machine Learning Era