Data Migration Lessons from the Trenches
summary
Migration moves many MongoDB clusters (about 30 source databases with wire versions from 3.2 to 3.6) into Amazon DocumentDB and reveals concrete blockers: DocumentDB lacks sharding and some index/array features, null bytes break restores, and maintenance windows can be tiny (about a 40 minute cutover window). The team uses custom Python sync tool that does dump/restore, pre-creates indexes, then replays oplog as upserts with namespace filtering and parallel workers, halting on errors for human triage and measuring remaining ops/rate to predict completion.
AI-written summary · about
chapters (11)
- 0:00Introduction and welcome
- 1:10Talk overview and project context
- 2:58Client perspective and project constraints
- 8:28Information gathering and coverage strategy
- 10:56Source database diversity and challenges
- 13:27Key differences between MongoDB and DocumentDB
- 17:03Migration strategy and preparation
- 21:40Custom sync tool development
- 27:39Parallelization and resumability
- 31:15Organizational communication and teamwork
- 34:44Q&A session
related talks
20mPercona docStreamer Migrating from Amazon DocumentDB to MongoDB
29mSwitching to MongoDB: The challenges of translating database schemas
53mManaging enterprise JSON workloads with Amazon DocumentDB
27mMurdering our Mongo Database (or how we migrated to a relational database)
24mMoving MongoDB to the Cloud: Strategies and points to consider
43mCRIMS: The case of the DB change that left no trace
21mSave, Grow and Innovate using AWS Databases
37mHow We Sharded and Migrated a 20 TB MongoDB Without Downtime