Planning for & Handling Failures from Open Hardware, Aviation, to Production at Google
summary
Practical rules and concrete failures: Google production uses revert first, review tags (TBR), a commit/continuous-integration queue and hardware-in-the-loop tests to catch integration bugs; rollouts go 1%, 5%, etc., because automation erased all drives once and flaky tests get disabled. Hardware examples include Samsung Note7’s multi-step battery failure and Boeing 737 MAX’s MCAS using a single AOA sensor plus missed training that helped cause crashes and ~$60+ billion in losses; emphasize blameless postmortems and rehearsed recoveries.
AI-written summary · about
chapters (27)
- 0:00Intro
- 7:31Learning by Example
- 9:22Samsung Note 7s
- 12:03Spares
- 12:16Other failures
- 16:22Change requests
- 17:08TBRS
- 17:25Unit Tests
- 17:53Continuous Integration
- 18:48Flakes
- 19:24Rollouts
- 20:28Postmortem
- 21:04Emergency Planning
- 22:06Online Commands
- 22:55Automation
- 24:14Postmoderns
- 24:25Have a Plan
- 25:28Aviation
- 27:36Automation in Aviation
- 28:51Automation in Cars
- 29:56Air France 447
- 32:59Automation is bulletproof
- 34:51Boeing 737 Max
- 38:25FAA
- 39:02Management Pressure
- 40:51The FAA
- 42:58Outro
related talks
50mLearning in Production (or Why the Apollo 11 Landing Nearly Failed)
39m10 Learnings From Running Production Infrastructure at Google
01h 06mCongratulations, you Failed!
15mWhy I Prefer Good Testing Over Excellent Testing
50mDevOps Project Failures - Lessons Learned and Strategies for Recovery
25mBrainstorming Failure
16mEngineering Failure-Resilient Systems
55mThe Troubles of Automating “all the Things”