Change Streams Don't Work
summary
Claim: MongoDB change streams fail to handle a large, write‑heavy sharded cluster reliably. Setup uses oplog, aggregation pipelines, resume tokens and fullDocument lookup against a cluster with 8 shards × 4 members = 32 data nodes, ~7 billion documents and ~128 TB storage. Built a resumable stream to mirror a dump/restore migration, but lookup overhead, resume window limits, batching latency and shard migration/orphaned‑document complexity make it lag and sometimes return empty results, so the migration approach does not work at scale.
AI-written summary · about
chapters (7)
- 0:00Start
- 15:25oplog.rs
- 30:18Event emitter
- 31:14while loop
- 31:24change event
- 33:24Resume token
- 35:14Filtering events
related talks
20mPercona docStreamer Migrating from Amazon DocumentDB to MongoDB
14mChange streams act on events in realtime
31mChange Data Capture for non relational databases
40mData Migration Lessons from the Trenches
30mChange Data Streaming Patterns in Distributed Systems
29mMulti Active-Active and disaster recovery with MongoDB database center
24mMoving MongoDB to the Cloud: Strategies and points to consider
29mSwitching to MongoDB: The challenges of translating database schemas