From concept to production; the fastest way to develop & deploy Apache Flink jobs
summary
Live demo shows Apache Zeppelin notebook tied to Kinesis Data Analytics for Flink, reading from Kinesis Data Streams and Kafka, joining with S3 lookups cataloged in Glue, and deploying to a managed Flink cluster on Kubernetes. Demo reads ~200k events quickly, computes aggregations with one-hour tumbling windows, visualizes results, and builds a runnable artifact to deploy to KDA. Scaling uses KPUs, checkpoints/savepoints persist state, and the end-to-end ETL runs successfully.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:10Demo
- 6:22Questions
- 12:00Schema
- 12:40Time window
- 14:30Running the job
- 14:57Data visualization
- 16:37Data catalog
- 17:43Union view
- 24:44Flink raw objects
- 27:30Flink visualization
- 28:02Flink job
- 28:54Summary
- 33:22Demo start
- 34:04Data generation
related talks
30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data Analytics
44mBuild a Real-time Stream Processing Pipeline with Apache Flink on AWS
46mFlink SQL in 2020: Time to show off!
26mRunning Apache Flink on AWS
38mApache Flink: A Very Quick Guide
43mA Data Streaming Architecture with Apache Flink #bbuzz
43mContinuous SQL with Kafka and Flink
22mDo Flink on Web with FLOW