Writing an interactive streaming SQL engine and pre-parser using Flink
summary
Claims that roughly 80% of streaming workloads map to SQL and shows a working interactive SQL editor plus backend engine called squealy o built on Apache Flink and Apache Calcite, using Kafka, Avro (everything converted to Avro), Kubernetes, WebSockets and a running Flink jar that accepts base64 payloads. It parses SQL, generates POJOs, samples 100 rows for interactive results, supports Kafka/S3/browser sinks and cross‑cluster joins, adds Nashorn JavaScript UDFs, and reports successes (rapid feedback, many short production queries) and hard problems (schema hygiene, read‑consistent sinks, autoscaling).
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:36Background and motivations
- 4:54Eventador SQL StreamBuilder
- 8:30Schema management - Virtual Table Registry
- 10:08SQL StreamBuilder Components
- 12:12Query Lifecycle - Parse
- 14:00Query Lifecycle - Execute
- 15:47SQL join streams from multiple clusters/types
- 17:40Building Processing Environments
- 18:52Javascript User Functions - Introduced Today
- 22:12Whats Next?
related talks
32mFlink's SQL Engine: Let's open the engine room!
23mAdvancing Data Mesh: Building A Stream Processing EcoSystem of Reusable Processors and Datasets
45mStream Processing with KSQLDB and REST API Queries
33mObservability Pipeline Query Languages: Present and the Future
38mBuilding Unified Streaming Platform at Uber
44mUsing Scala.js and functional reactive programming for developing (enterprise) applications
47mStream All the Things!
45mSQL from the Frontend: Viable or Not?