Spell of Transmutation - Using Debezium to Transfer Realtime Data Changes to Event Hub

on YouTube ↗46m

" summary

Sets up Debezium to stream SQL Server CDC into Azure Event Hub using an Azure Container Instance that posts via the Kafka endpoint; includes a local Azure Function and Stream Analytics consumer. Covers exact config choices: snapshot mode vs initial, need for a persistent history store, disabling tombstones on delete false, and SQL firewall/Kafka gotchas. Warns container constantly polls (so serverless SQL wakes and costs spike). Demos are recorded and show inserts flowing end-to-end.

AI-written summary · about

" chapters (13)

1. 0:00Introduction to Debezium
2. 4:44Why use near real-time
3. 6:46Understanding the components
4. 11:34Getting started with Debezium
5. 14:21Preparing the SQL database
6. 18:48Setting up the event hub
7. 23:50Deploying the container
8. 27:25Configuring the connector
9. 31:19Connector gotchas and tips
10. 36:05Verifying and debugging
11. 37:43Consuming the data stream
12. 40:18Using stream analytics
13. 43:15Conclusion and best practices

" related talks

• 01h 11mChange data capture pipelines with Debezium and Kafka StreamsGunnar Morling · Joker 2020
• 30mChange Data Capture With Flink SQL And DebeziumMarta Paes · DataEngBytes 2020
• 26mDebezium Server - New CDC runtime on the horizon!Ondrej Babec, Jiri Novotny · SREday London 2024
• 48mStreaming Database Changes with DebeziumGunnar Morling · Devoxx Belgium 2017
• 31mChange Data Capture for non relational databasesJakub Cechacek · DevConf.CZ 2023
• 27mUnderstanding stream table duality using Kafka, Flink SQL and Debezium formatMuhammet Orazov · Community Over Code EU 2024
• 47mBuilding highly scalable IoT streaming solutions using Azure Event Hubs and MongoDBRobert Walters · Global Summer 2020 Azure Data Fest 2020
• 43mImplement a Serverless Data Platform with Microsoft AzureRoberto Freato · Big Data Conference Europe 2023