Incident Response @ FB, Facebook's SEV Process
summary
Explains Facebook’s incident workflow: on‑call discovers incidents, escalates to an imoc who coordinates (not fixes), uses IRC logs, Messenger and a centralized sev manager for timelines and action tracking, and biases toward over‑classifying severity. Reviews happen twice — team deep dives and a weekly production review for cross‑team learning — with blameless culture and tracked follow‑up bugs. Gives a concrete failure where a rollback command caused a global canary restart, and shows how one review stopped at least four other teams from repeating the same mistake.
AI-written summary · about
chapters (23)
- 0:00Intro
- 0:45Welcome
- 1:35Failure is a matter of when
- 1:54Facebooks reach and scale
- 2:56What Im sharing today
- 3:57An anecdote
- 4:55The virtuous cycle
- 6:38Incident management patterns
- 8:12How Facebook manages incidents
- 12:25SEV Tools
- 14:08SEV Manager
- 17:40Antipatterns
- 19:03Review
- 20:30Culture
- 21:58Production Review
- 22:40Regular cadence
- 23:38Three questions
- 24:35Sharing knowledge
- 25:13What next
- 26:29Data collection
- 27:47Real story
- 30:55Summary
- 31:49QA