Handling a Major Outage Transparently
summary
GitLab suffers an accidental production DB deletion that loses six hours of data for issues, comments and merge requests but spares repositories; incident response posts a live Google Doc and runs a public livestream that peaks at 5,000 viewers and tops Hacker News while earning ~3,000 supportive tweets. Identity of the person who caused it remains protected, no firing occurs, and followups include switching to Wally, automated backup testing, better docs and an owner for backups.
AI-written summary · about
chapters (7)
- 0:00Start
- 0:54Source code management • Issue tracking • Code review . Continuous Integration (running the test suite)
- 5:56Protect the identities of anyone involved
- 7:34Don't distract engineers working on fixing the problem.
- 8:38Work in shifts, don't have any one person working for more than a few hours at a time.
- 8:50What we learned Audit your backups and procedures for running commands on production servers • Have someone who owns the backups • Communication with users, even in really bad situations, leads to better outcomes
- 11:28We did not fire the engineer involved in this mistake.
related talks
27mWhen One Line Took Thousands of Websites Offline
35mGitOps or a Journey to a Production Incident and Swiftly Back Again
51mDatabase Incident Management: When Everything Goes Wrong
22mAfter the dust has settled: learning from incidents
22mWhat can possibly go wrong? The story of an outrage at Booking.com
22mOops, I deployed too hard
46mFrom 1,000 to 5,000 Users: Scaling GitLab Without (Too Much) Pain
38mHow We Scaled GitLab For a 30k-employee Company