Runway Overrun
summary
An engineer types a command and a production database dropped at DigitalOcean (5 Apr 2017, 10:24 ET), taking the company offline for five hours and requiring restore from an async replica and backups; investigation shows missing guardrails (developer laptops with direct prod access) and a Rails change later prevents this. A separate incident (American Airlines 2253, 28 Dec 2010) sees speed brakes failed to deploy and a 10‑microsecond thrust‑reverser window, a <1‑in‑a‑million failure; aviation fixes are training, memory actions and CRM rather than blaming individuals. The recommendation: analyze systems, add safety scaffolding and remove ways to accidentally touch production.
AI-written summary · about
chapters (17)
- 0:00Conference opening
- 1:13Welcome to RubyConf
- 2:37Computers and system failures
- 4:05Production database disaster
- 6:12Runway overrun incident
- 8:42Blame vs system analysis
- 10:12Dreyfus model of skill
- 12:58Pilot expertise and automation
- 13:40How planes slow down
- 16:52System reliability limits
- 17:56NTSB recommendations
- 19:40Pilot training improvements
- 22:14DigitalOcean incident analysis
- 23:40Root cause fallacy
- 24:43Engineering team responsibility
- 25:47External failure analysis
- 26:30Closing thoughts
related talks
31mOperating Rails: What About After You Deploy?
32mOperating Rails: what about after you deploy?
23mSquash Production Defects Quickly - The Power of Structured Logging in Rails
34mBeyond CRUD: the PostgreSQL techniques your Rails app is missing.
34mIndustrial Rails
31mTesting in Production
40mThe Life and Death of a Rails App
40mGrowing internal tooling from the console up