Uber's Failover Architecture: Reconciling Reliability and Efficiency in Hyperscale...
summary
Claims a shift from uniform 2× regional provisioning to a differentiated model that cuts steady‑state from 2× to 1.3×, raises utilization from ~20% to ~30%, and sustains ~99.97% availability; deploys automated orchestration and safeguards — Outage Mitigator (OMG) for parallel failover steps, Kraken for prefetching images, RPC shield plus a canary regression gate to eliminate fail‑close dependencies; hardens ~4,000 unsafe dependencies, reclaims ~575K cores (projected >1M from ~4M), and restores full capacity in ~7 minutes with steady state reached in ~20–30 minutes.
AI-written summary · about
chapters (11)
- 0:00Uber's failover architecture
- 1:21Dual-region active-active challenges
- 3:07Failover goals and tiers
- 6:38Outage Mitigator and Kraken
- 9:11Image pre-fetching and migration
- 11:22Resource borrowing and buffer
- 13:04Eliminating fail-close dependencies
- 14:51Testing and regression systems
- 16:16Real-world failover results
- 17:39Error rates and utilization impact
- 18:41Lessons learned and future
related talks
23mDevOps at Uber
23mKeynote: Software-Defined Infrastructure at Uber Scale
32mBeyond Optimization: Resilient Cloud Microservices at Scale
16mFrom System TAYE7 to High Availability: What, Why, How?
19mZero-Downtime by Design: High-Availability Migration Patterns
43mGoogle - Load-shedding
56mMastering Cloud Scalability and System Designs Deep dive
25mTackling operational and capacity utilization challenges at a large scale enterprise deployment
