Loading into Kafka from a Database: With and Without CDC
summary
Using Kafka Connect to stream database changes into Kafka both with CDC (WAL/log-based) and without it, the talk examines practical problems like master node failures, extracting data when WALs or timestamps are unavailable, and writing a custom connector when OSS options fall short. It also covers enterprise operational concerns: unifying metadata, CI-based deployment and rollback, secret management, and lessons from building an internal PaaS used by dozens of teams processing thousands of messages per second.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:07Streaming platform and Kafka
- 5:15Why load into Kafka
- 7:22First hypothesis: DB writes and Kafka
- 9:51Debezium CDC: how it works
- 11:37Data loss on master failure
- 12:38Connecting to specific host
- 13:43Risk of breaking DB
- 14:46Adding new tables
- 15:50Oracle GoldenGate issues
- 16:52CDC alternative: microbatches
- 18:16Microbatch modes
- 21:07Custom connector without CDC
- 22:12Building custom connector
- 25:23Deploying connectors in production
- 26:48Monitoring connectors
- 27:50Schema evolution and duplicates
- 29:14Summary and contacts
- 29:36Q&A
related talks
27mNo More Silos: Integrating Databases and Apache Kafka
18mBuild a data pipeline in 20 minutes with Kafka Connect
49mLet’s Build a Streaming Data Pipeline!
43mBack to fundamentals: from databases to event-driven platforms
30mChange Data Streaming Patterns in Distributed Systems
39mDeep Dive on Using YugabyteDB's Change Data Capture Connector
29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data Pipeline
41mStreaming your database - Easier said than done?