Writing a Streaming SQL & Materialized View Engine
summary
An interactive SQL engine built on Apache Flink and Apache Calcite parses statements with a SquealyIO parser, compiles POJOs into a Flink job, and samples results back over WebSockets into the Python/JS console while Kafka and RocksDB handle state; a Snapper materialized views service persistently stores latest state by key, exposes JSON REST endpoints, supports queries by any predicate (not just primary key), uses JS input transforms compiled to JVM bytecode, runs on Kubernetes, and the demos show it working end to end.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 0:35Background
- 4:20Sequel Stream Builder
- 7:05Sources and Sinks
- 12:18Running Sequel
- 18:54Materialized Views
- 25:28Flink
related talks
23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
41mStreaming SQL
21mStream Processing and SQL
32mThe easy way to build and run Flink SQL jobs using Cloudera SQL Stream Builder
43mUnified stream and batch processing with Apache Flink’s relational APIs
22mStream SQL with Flink @ Yelp
48mUnlocking Real-time Insights: Enhancing Your Databases With Stream Processing
23mFrom Streams to Tables and Back Again: A Demo of Flink’s Table & SQL API