Flink's SQL Engine: Let's open the engine room!
summary
Walks the SQL string through parsing, validation, rule-based and cost-based rewrites, physical planning and mapping to Flink stream execution nodes, showing how watermarks are validated and pushed into sources, how timers and state are created for an interval join, and how change log normalize turns Kafka records into insertion/update/delete streams. Shows how upsert keys and changelog modes traverse topology, uses Kafka and CDC connectors, produces exactly-once Kafka sinks, supports savepoints, and runs on managed Flink with autoscaling and scale-to-zero.
AI-written summary · about
chapters (7)
- 0:00Introduction to Flink
- 1:25Stream processing basics
- 6:23What is Flink SQL
- 15:18SQL planning engine
- 21:39Physical execution plan
- 28:21Flink SQL in practice
- 30:48Q&A and closing
related talks
31mWriting an interactive streaming SQL engine and pre-parser using Flink
37mKeynote | Flink First: Designing Modern Trading Systems
37mNot So Big – Flink as a true Application Framework
38mBuilding Unified Streaming Platform at Uber
23mAdvancing Data Mesh: Building A Stream Processing EcoSystem of Reusable Processors and Datasets
43mStructure and Interpretation of Stream Processing
29mStreaming & Eventing for Top Management
36mWhen ordering matters