Data Migration With Confidence
summary
Migration moves millions of rows and 100k+ images from legacy MySQL dumps into a Rails schema using ActiveRecord migrators, transactions, retry-on-deadlock, Redis caching and Graphite/Grafana for CPU/IO monitoring. Strategies include editing dumps with mysqldump and hex editor, bulk import via load data infile or active_record_import, throttling workers to avoid IO saturation, pre-uploading images and reusing deterministic image IDs so uploads take 10 days for 100k photos instead of blocking migration, and a lazy per-user migration that runs on login to avoid a massive upfront run.
AI-written summary · about
chapters (48)
- 0:00Intro
- 1:20Data Migration?
- 1:43Transfer data from System, to System
- 2:08Data Migration to Existing System
- 2:24Why Migration?
- 2:55Get All Data to our system
- 3:12Provider API Data Dump
- 3:55Start with a rake task
- 6:16Modeling Database
- 6:40Map data to your current system
- 7:31Setup Test Suite
- 8:08Better code through boring tests
- 8:34Use all methods that raises exception
- 8:46Fail Fast to find all errors
- 10:10Data Integrity
- 10:28Idempotent Operation
- 10:42Run Migration many times
- 11:25Accuracy
- 11:46Example Check users with most Recipes
- 12:58Add more Checker objects
- 13:14Objects Everywhere
- 14:16Log Every Unexpected Error
- 14:48Fix every error you can before real migration
- 14:58Retry mechanism
- 15:22MySQL deadlock
- 16:59Status Reporting
- 17:52Monitoring CPU Usage
- 18:17Performance is a Rabbit hole
- 18:39Preload associations
- 18:56Avoid unnecessary callbacks
- 19:27Process multiple records in one job
- 20:01Migrate Important things first
- 20:34Scale up the Database
- 21:12Every change to make it fast
- 21:35Update all necessities
- 22:28Stories
- 23:42Migrate Millions of records
- 24:31Migrate their data in high priority
- 25:11Migrate 100K photos
- 26:10Set the designated hash during migration instead of upload, generate hash
- 26:20Benchmark how long to finish all
- 26:3599% photos won't change
- 26:48Migrate users password to secure auth
- 27:13System's password auth will fail
- 27:28Fallback to Legacy Auth
- 27:44When password matched from legacy auth
- 28:07Migration done in Ghost Table fashion
- 28:21Data Dump & Generic Migration Code
related talks
37mData Migration With Confidence
59mLarge Scale MySQL Migration to PostgreSQL
44mMongoDB to MySQL the How and the Why
40mrails db:migrate:safely
24mMigrations under Production Load: How to Switch...
29mScaling Rails to 2M MySQL requests per second
37mDenormalizing Rails
38mEfficient Reliable Database Migration: A Legacy Innovation Story