Change Data Capture With Flink SQL And Debezium

on YouTube ↗30m

" summary

Flink 1.11 consumes Debezium JSON changelogs from Kafka and Flink SQL deserializes them directly, proven with a Docker Compose demo wiring Postgres → Debezium (Kafka Connect) → Kafka → Flink SQL → Elasticsearch/Kibana. The demo bootstraps 1000 rows, shows insert/update/delete captured end-to-end, builds a materialized aggregation by insurer and animal, and updates the Elasticsearch index in real time while a datagen inserts records every 0.5s. Production scale cited: 10,000 Flink SQL jobs, 2.5 billion events/sec, 100 TB state.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 1:58What is Change Data Capture
3. 4:28Querybased Change Data Capture
4. 8:07Logbased Change Data Capture
5. 14:46Demo Setup
6. 15:28Testing
7. 21:07Flink SQL
8. 25:42Kibana
9. 27:18Wrap Up

" related talks

• 42mChange Data Capture and Processing with Flink SQLJark Wu, Qingsheng Ren · Flink Forward Global Virtual 2020
• 43mContinuous SQL with Kafka and FlinkTimothy J Spann · JCON WORLD ONLINE 2023
• 27mUnderstanding stream table duality using Kafka, Flink SQL and Debezium formatMuhammet Orazov · Community Over Code EU 2024
• 46mFlink SQL in 2020: Time to show off!Timo Walther, Fabian Hueske · Flink Forward Virtual 2020
• 39mChangelog Stream Processing with Apache FlinkTimo Walther · Berlin Buzzwords 2022
• 01h 11mChange data capture pipelines with Debezium and Kafka StreamsGunnar Morling · Joker 2020
• 48mStreaming Database Changes with DebeziumGunnar Morling · Devoxx Belgium 2017
• 42mFlinkCDC: Streamlining your data analytics pipelinesMuhammet Orazov · Berlin Buzzwords 2025