Too Big to Fail
summary
Company transfers hundreds of millions (soon billions) so risk is concrete: Ruby on Rails code runs validating every model in production every three hours to catch corrupt or invalid records, a library called Broken Record flags problems, and a runner named Ultramarathon breaks a monolithic maintenance job into quarantined steps with explicit dependencies and instrumentation. Partial failures persist state instead of restarting everything, recovery becomes automated, and early results pay off despite long run times.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 0:52Agenda
- 1:50Robert Caro
- 7:54Precogs
- 8:50Long Running Jobs
- 10:47Recovery
- 11:24Mia Hamm
- 11:49Back in time
- 12:20System of Maintenance
- 13:48Brain Surgery
- 14:04Dependencies
- 15:04IRS Shutdown
- 16:39Rewriting Rails
- 16:54Hacker News
- 18:02Maintenance
- 18:26Grouping
- 19:28Ultra Marathon
- 20:12Life Cycle
- 20:45Running Blocks
- 21:42Instrumentation
- 22:53Accepting Failure
- 23:08Failure is Not an Option
- 25:32Quotes
- 27:09Misconceptions
related talks
41m(Re)Building a large scale payments system on Rails
28mPerfect is too expensive
39mData Corruption: Stop the Evil Tribbles
34mOh S***: How to bring a big Rails website down (and how not to)
48mFault Tolerance on the Cheap Making Systems That Probably Won't Fall Over
42mPlanning To Fail
32mBuilding for Gracious Failure
31mOperating Rails: What About After You Deploy?