Data Pipeline Lifecycle: SQL Everywhere

on YouTube ↗31m

" summary

SQL from Presto is reused verbatim in Flink SQL to turn a historical analyst query into a streaming pipeline that joins a Kafka four-message toy dataset with Postgres pricing via a lookup, adds tumbling windows with watermarks and correct timestamp precision, and writes JSON results to a Kafka topic. The pipeline compiles into a Flink job, runs (staging shows ~40 messages produced), and is backed by file-based integration tests before expanding to datastream API.

AI-written summary · about

" chapters (12)

1. 0:00Intro
2. 0:21Humans need programming languages
3. 2:13Python
4. 4:28Business Use Case
5. 6:23Data Set
6. 6:58Analyst
7. 7:27User Behavior
8. 7:54Collaborate
9. 11:44Aggregation
10. 19:57Connecting Kafka
11. 22:20Testing in Scala
12. 26:05Integration test

" related talks

• 43mLeverage the power and simplicity of SQL on Apache FlinkFabian Hueske · Berlin Buzzwords 2018
• 23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · DataEngBytes 2021
• 43mContinuous SQL with Kafka and FlinkTimothy J Spann · JCON WORLD ONLINE 2023
• 39mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · Data Con LA 2021
• 46mFlink SQL in 2020: Time to show off!Timo Walther, Fabian Hueske · Flink Forward Virtual 2020
• 39mChangelog Stream Processing with Apache FlinkTimo Walther · Berlin Buzzwords 2022
• 42mChange Data Capture and Processing with Flink SQLJark Wu, Qingsheng Ren · Flink Forward Global Virtual 2020
• 43mA Data Streaming Architecture with Apache Flink #bbuzzRobert Metzger · Berlin Buzzwords 2016