Delete This: Decommissioning Servers at Scale

on YouTube ↗25m

" summary

Facebook automates server decommissions with a migration handler framework plus a UI and ticket state machine that take retiring and replacement sets, spawn handlers, and fall back to on-call tickets; it adds a returned state in hardware tracking and a destructive wiping service nicknamed robot butcher to finish decomm. Running in prod for over a year, this pipeline reclaims idle capacity for stateless jobs, cuts human tickets and errors, but initially wiped live cache machines due to a handler bug and was fixed with the new state and clearer tickets.

AI-written summary · about

" chapters (4)

1. 0:00Start
2. 1:09Annie Rhodes
3. 3:56Service Migration
4. 9:47The Migration Handler Framework Prototype

" related talks

• 19mDelete This: Decommissioning Servers at ScaleAnirudh Ra · SREcon18 Europe/Middle East/Africa
• 19mScaling Back-End DeploymentTyler Petrochko · Systems @Scale Fall 2021
• 41m"A Day in the Life of Facebook Operations"Tom Cook · Velocity 2010
• 40mA Customer Service Approach to SREJohn P. Looney · SREcon19 Europe/Middle East/Africa
• 34mZero Downtime Database Migrations and DeploymentsOndrej Mirtes · Dutch PHP Conference 2018
• 40m"Removing the Human SPOF"Justin Huff, Tim Wilfong · Velocity 2010
• 16mDeleting Data at Organizational ScaleDiogo Lucas · PEPR '24
• 29mTaming the Hybrid Beast with DevOpsGary McKay, Wesley Pullen · DevOps Enterprise Summit London 2018