How we Migrated (...) Without Downtime using Elixir
summary
An Elixir-based migration service that dual-writes new records to S3 and the legacy database, then uses a Broadway pipeline to read the old DB, track progress via persistent Redis offsets, apply a simple transform (remove leading underscore from id and drop the revision field), and write batched objects to S3. With up to 100 concurrent writers and tuned batch sizes they achieve ~450k writes/sec and migrated about 1 billion objects (~500M estimates + ~400M final fares) in ~two weeks with ~99.99% success and no downtime.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:27Cabify’s business and tech
- 3:54Elixir in pricing
- 8:17Legacy data issues
- 10:04Migration to S3
- 11:29Elixir migration service
- 14:03Broadway for concurrency
- 16:56Pipeline architecture and perf
- 19:07Migration results and timeline
- 21:22Benefits of Elixir
- 25:23Q&A: Testing
- 28:46Q&A: Race conditions
- 29:13Q&A: Legacy shutdown
- 30:03Q&A: Migration timing
- 31:29Q&A: Reusability
- 34:46Q&A: Data storage
- 36:41Closing
related talks
25mMigrating from C# to Elixir
31mMigrating a Warehouse Application from C# to Elixir
47mElixir for curing product development pains
24mMigrations under Production Load: How to Switch...
49mFrom Legacy To Event Sourcing
47mElixir for curing product development pains?
37mFrom the ground to the stars
53mHow to migrate a mission critical railway system