Change Data Capture and Processing with Flink SQL
summary
Flink SQL 1.11 implements FLIP-95 so sources can emit insert/update/delete changelog and the pipeline reads snapshots then binlogs; a demo wires MySQL/Postgres CDC connectors (Debezium engine embedded), joins three source tables, and writes to Elasticsearch and Kafka using a changelog JSON format. The demo starts with three orders, shows live inserts, updates and deletes, materializes GMV changes (50.5 → 65.5 → 90.75 → 140.75 etc.), and the end-to-end pipeline works using pure SQL.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 0:31Agenda
- 1:03Data Warehouse Architecture
- 2:29Change Data Capture
- 6:03Logbased CDC
- 7:52Demo
- 10:32Changelog Mechanism
- 13:26Flink 95
- 17:47Flink Sequel
- 20:09CDC Streaming Pipeline
- 20:54Demo Intro
- 22:06Demo Environment
- 23:51Demo Start
- 26:42Create Source Tables
- 28:04Execute the Job
- 28:39Job topology
- 31:52Update status
- 33:36Create new sync table
- 37:28Insert new order
- 37:57Reset price
- 39:39Summary
- 40:27Future plans
related talks
30mChange Data Capture With Flink SQL And Debezium
39mChangelog Stream Processing with Apache Flink
43mChange data capture in production with Apache Flink
43mContinuous SQL with Kafka and Flink
42mFlinkCDC: Streamlining your data analytics pipelines
46mFlink SQL in 2020: Time to show off!
27mUnderstanding stream table duality using Kafka, Flink SQL and Debezium format
23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL