Streaming SQL

on YouTube ↗41m

" summary

Apache Calcite extends SQL with a stream-table duality so the same object can be queried as a growing table or as a continuous stream; watermarks and an omit clause let queries make progress and handle late arrivals. Flink uses Calcite for parsing and cost-based planning, supports tumbling/hopping/session/sliding windows, and requires time constraints for stream–stream joins. Queries can feed BI tools like Tableau and the planner can replan at runtime (e.g., after minutes of sampling if throughput is 10/sec vs 10M/sec).

AI-written summary · about

" chapters (17)

1. 0:00Intro
2. 1:33What is streaming
3. 2:19Why are people streaming
4. 5:03Why query streams
5. 10:10Database workloads
6. 11:35Sequel extensions
7. 15:21Stream table duality
8. 16:30Why is it important
9. 19:54Semantics
10. 23:33Handling Late Data
11. 27:25GroupBy
12. 27:52Window Types
13. 28:59Pie Chart Problem
14. 30:47Joining Streams
15. 31:30Calcite
16. 32:24Apache
17. 33:22Questions

" related talks

• 21mStream Processing and SQLBruno Bonnin · ScalaIO 2018
• 43mUnified stream and batch processing with Apache Flink’s relational APIsFabian Hueske · Flink Forward Berlin 2017
• 23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · Berlin Buzzwords 2021
• 31mWriting a Streaming SQL & Materialized View EngineKenny Gorman · Flink Forward Virtual 2020
• 39mBuilding modern SQL query optimizers with Apache CalciteVladimir Ozerov · ApacheCon@Home 2021
• 58mApache Calcite: A Platform for Building Advanced SQL Optimizers on JavaVladimir Ozerov · Joker 2021
• 40mOne SQL to Rule Them AllFabian Hueske · Flink Forward Europe 2019
• 40mThe Seven Deadly Sins of StreamingGiannis Polyzos · Big Data Conference Europe 2025