A deep dive into Flink SQL
summary
Explains how Flink SQL unifies batch and streaming via a pluggable blink planner that becomes default, translates Table API into a logical plan, uses Apache Calcite rules (expression reduction, filter and projection pushdown) and picks physical operators (e.g. broadcast hash join when one side is ~1,000 rows). Describes a compact binary row layout to avoid full deserialization and claims >50% throughput gains, changelog encoding of insert/update/delete for correct cascaded aggregations, plus mini-batch and local pre-aggregation to cut state I/O.
AI-written summary · about
chapters (18)
- 0:00Intro
- 1:08Architecture before Flink 1.9
- 1:58A Step Closer
- 3:05Future Architecture
- 3:26Architecture since Flink 1.9+
- 4:45How Flink SQL Works?
- 7:07An Example
- 7:58Expression Reduce
- 9:23Filter Push Down
- 10:26Projection Push Down
- 11:23Physical Planning (Batch)
- 12:40Translation & Code Generation
- 14:12Physical Planning (Stream): Changelog Mechanism
- 23:57Old Planner: Row
- 28:36Mini-Batch Processing
- 30:37Aggregation Skew Handling
- 32:47Plan Rewrite (Top-N)
- 35:59Summary & Futures
related talks
47mTaking a look under the hood of Apache Flink’s relational APIs
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
42mApache Flink deep-dive #bbuzz
25mContinuous Queries on Apache Pulsar using Flink SQL
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
44mFlink’s New Batch Architecture
38mFaster and Furiouser: Flink Drift
43mStream Processing and Applications in the Modern Age