Multi Active-Active and disaster recovery with MongoDB database center
summary
Built a cross-region replicator called Lamda with manager, collector, tunnel and receiver that streams oplog entries between clusters, uses a global ID to prevent loops, and integrates Kafka-backed tunnel and CRC32 verification plus snappy/gzip compression. Deployed tens of pipelines between data centers hundreds of miles apart, handling ~2 billion oplog entries per day (>10 GB incremental, 70% updates), adaptive batching and async transfers, parallel replay by collection/document, and achieves RPO <1s RTO <1min for a three-region map application.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:20About me
- 1:18Outline
- 5:54Replication
- 8:47Double Active Replication
- 10:07Lambda Replication
- 14:49Adaptive Batching
- 17:11Parallel Operations
- 21:47Business Case
- 26:02Quality of Date
related talks
38mDisaster Recovery with a Distributed Database
43mMulti-Cloud Magic: Leveraging Multi-Cloud Clusters in Real-World Scenarios
31mDeep dive on cross-Region DR with Amazon Aurora global databases
40mMulti Cloud Magic: Leveraging Multi Cloud Clusters in Real World Scenarios
54mBreakout Session
12mFault-Tolerant Replication with Pull-Based Consensus in MongoDB
11mZero Data Loss at Scale
31mBuilding a global-scale data platform with Kubernetes, MongoDB, and HashiCorp tools