How to Migrate 50PB into 32PB Without Downtime?
summary
A Mail.ru Mail engineering team describes designing a new attachment storage system with deduplication (architecture and safeguards), explains how deduplication saved about 18 PB, and details an in-place migration method to move 50 PB into the new layout without downtime.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:04Email data structure
- 15:30Deduplication system design
- 22:15File upload process
- 28:04File deletion and consistency
- 37:58Migration results and savings
- 38:42Q&A
related talks
23mHow Atlassian is migrating 40PB data with 99.99% reliability
37mHow We Sharded and Migrated a 20 TB MongoDB Without Downtime
26mImproving Your Services the DevOps Way
44mA case study on building a mail archive with postfix
31mModernizing a megascale database
32m"From On-Prem to the Cloud and Back Again"
31mZero Downtime Migrations at Scale
44mEmail at Scale: How We Survived 800M Emails/Year (and DNS)