The worst outage I never caused
summary
A mistaken config change nearly stops most of Google from being advertised to the internet by removing a critical routing flag, which would have taken effect globally in 2–3 seconds. Investigation finds unowned Juniper-style configs, missing tests, and about ~200 lines commented out; verification relies on protocol buffers. Response builds unit checks that assert >100 routes, uniqueness, and magic-number correctness and a Python generator that emits validated configs so changes become safe and reviewable.
AI-written summary · about
chapters (6)
related talks
18mThe worst outage I never caused
29mThe worst outage I never caused
40mThe World Blew Up but We’re All Okay: How We Managed a...
34mHow We Drained Every Backbone Router Simultaneously
39mOops. I broke the Google, now what?
47mExtending SDN to the Management Plane
15mQUIC Exit: Exposing a New Class of Outage
22mWhy is it always DNS, TLS, and bad configs?