Restoring FDB after Catastrophic Accidents
Talkat FoundationDB Summit 2019 · on Monday, 2019-11-18· at 14:10· Grand Ballroom 5, Marriott Marquis San Diego Marina·
summary
A multi-petabyte FoundationDB deployment recovers from three real incidents using FDBDR, Linux filesystem snapshots and EBS snapshots, proving an end-to-end restore of a 20 TB mirror after kernels were wiped and verifying the process on replayed production traffic. Recovery uses automated runbooks, LVM restore of boot volumes, coordinator tagging, and a scripted fleet replace to bring new instances up. Lessons: 20 terabytes gone is recoverable, EBS snapshots work but are slow, and FDBDR yields fast catch-up when prepped.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:51FoundationDB scale
- 3:15Missing kernels disaster
- 8:35Lessons learned
- 11:21Restoring FDB from EBS
- 13:09Summary & Q&A
related talks
34mFoundationDB, the Black Knight
26mRunning FDB at scale at Wavefront
51mDatabase Disasters and How to Find Them
27mBackup and Restore: Discover Best Practices for a Distributed Database Backup Strategy
22mHow FoundationDB powers SnowflakeDB's metadata!
22mPostgreSQL Disaster Recovery Fundamentals
47mDisaster Recovery Strategies for SQL Server
39mDon’t eat my data - 30+ years of storage war stories