Changelog Stream Processing with Apache Flink

on YouTube ↗39m

" summary

Apache Flink SQL processes tables as change logs and ships an optimizer that turns streams into Flink SQL as changelog processor semantics: four change kinds (insert, update-before, update-after, delete) flow through operators. Demo uses MySQL with Debezium/Flink CDC versus JDBC snapshot, joins transactions to customers with a time-version lookup join, and shows a upsert optimization that cuts downstream traffic by ~50% when primary keys exist. Table API can export to DataStream and back for custom operators or ML.

AI-written summary · about

" chapters (16)

1. 0:00Introduction
2. 0:41Who am I
3. 1:28What is Apache Flink
4. 4:02How Flink works
5. 6:01Who is using Flink
6. 6:43Flink APIs
7. 7:49Data Stream API
8. 9:12Table and SQL API
9. 12:41Flink SQL
10. 13:52Stream Table Duality
11. 15:41Stream Table Duality Example
12. 17:51Sources
13. 19:34Retraction mode
14. 24:46Transitions
15. 26:11Demo
16. 33:58Summary

" related talks

• 42mChange Data Capture and Processing with Flink SQLJark Wu, Qingsheng Ren · Flink Forward Global Virtual 2020
• 27mUnderstanding stream table duality using Kafka, Flink SQL and Debezium formatMuhammet Orazov · Community Over Code EU 2024
• 23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · DataEngBytes 2021
• 35mFlink's Table & DataStream API: A Perfect SymbiosisTimo Walther · Flink Forward Global 2021
• 30mChange Data Capture With Flink SQL And DebeziumMarta Paes · DataEngBytes 2020
• 39mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · Data Con LA 2021
• 43mLeverage the power and simplicity of SQL on Apache FlinkFabian Hueske · Berlin Buzzwords 2018
• 23mFrom Streams to Tables and Back Again: A Demo of Flink’s Table & SQL APITimo Walther · Flink Forward Berlin 2017