How We Drained Every Backbone Router Simultaneously
summary
A routine capacity command took all eight EBB planes offline and a simultaneous DNS heartbeat failure causes every name server to withdraw, effectively blackholing traffic; recovery requires physical technicians to log into routers and flip them, and the outage lasts four to four and a half hour outage. Restoring service is throttled because traffic re-entry would cause a 10 megawatt dip and then surge, so teams use overload protection/defcon tools and staged bring‑backs, plus postmortem fixes to audits and safety checks.
AI-written summary · about
chapters (5)
related talks
34mHow We Drained Every Backbone Router Simultaneously
04mLoad balancing through the years
34mThe Day the DNS Died
24mBlink: Fast Connectivity Recovery Entirely in the Data Plane
23mAutomatic Traffic Scheduling for Internet Connectivity Failures
18mThe worst outage I never caused
29mUsing the Internet as Your Load-Balancer
38mLogs Told Us It Was DNS, It Looked like DNS, It Had to Be DNS, It Wasn't DNS