Change Data Capture for non relational databases
summary
Debezium connects MongoDB replica sets or sharded clusters to Kafka (or Kinesis, Pub/Sub, HTTP) by taking a snapshot then streaming change streams using a resume token stored as offsets; resume is limited by the oplog retention so outages longer than that force a full snapshot. For sharded clusters Debezium can open per-shard streams for higher throughput (per-document ordering only) or a mongos stream to order across shards with added latency. Outbox pattern: write an order and an outbox collection row in one DB transaction and Debezium routes that event to Kafka; delivery is at-least-once (Kafka can provide exactly-once semantics).
AI-written summary · about
chapters (29)
- 0:00Introduction
- 0:10Agenda
- 2:05Alternatives
- 3:00Use Cases
- 3:40What is the Division
- 4:00Division Connectors
- 5:25Kafka connectors
- 7:45MongoDB
- 10:50Operations Law
- 12:20Deployment for MongoDB
- 12:55sharded clusters
- 13:15horizontal scale
- 14:05chain streams
- 17:35replica sector collection
- 19:40practical problems
- 19:50simple order service
- 20:10dual rights
- 21:00outbox pattern
- 21:20work worker process
- 21:55order services
- 22:10process
- 23:50dbzoom
- 25:10Cluster setup
- 25:40Configuration
- 27:35mongo shell connection
- 27:55mongo shell simulation
- 28:30Kafka
- 30:10Summary
- 30:40Questions
related talks
01h 09mOpen-source Change Data Capture With Debezium
49mChange Data Capture Pipelines with Debezium and Kafka Streams
41mChange Data Streaming Patterns in Distributed Systems
48mStreaming Database Changes with Debezium
23mChange Data Capture: A RabbitMQ and Debezium Approach
01h 11mChange data capture pipelines with Debezium and Kafka Streams
42mData Streaming for Microservices using Debezium
19mChange data capture with Debezium…and without