Embracing Failures to Avoid Meltdown
summary
Company runs a Ruby on Rails service that checks 100,000 keywords for 10,000 customers nightly; spikes of 500 errors occur when the third‑party ranking API slows and the app retries every job. Engineers implement a circuit breaker wrapper (simple Ruby class or a Netflix‑style gem) with timeouts, a 5‑failure or percentage threshold, fail fast semantics and a half‑open probe. Breaker metrics gate deployments, enable cache/backup fallbacks, and reduce downstream load so the system recovers without midnight outage calls.
AI-written summary · about
chapters (9)
- 0:00Introduction and welcome
- 1:22Why failures are inevitable
- 5:26What is a circuit breaker
- 9:49Real-world failure example
- 13:26Fail fast and timeouts
- 17:51Circuit breaker states
- 21:46Monitoring and automation
- 23:56Implementing in Ruby
- 25:54Q&A
related talks
32mBuilding for Gracious Failure
42mThe world runs on bad software
39mMorning Keynote: Embracing an Error-Prone Reality
31mFuture Proofing your Ruby Stack
44mThe role of catastrophic failure in software design
31mWe All Make Mistakes: Learning from Gaffes in the Ruby Community
41m‘Failure’ as Success: The Mindset, the Methods, and the Landmines
04mpreventing Cover-Your-Ass engineering