Infrastructure: Engineering for Resiliency at Scale
summary
An unexpected December 16, 2021 event causes ~30 minutes of slowed customer requests when centralized UPS rectifiers fail to track a noisy utility, triggering IGBT d-sat causing interrupted bypass transfers and dropped load. Infrastructure evolves from ~2 MW lineups (70 racks) to 15 kW single-rack blast radius with rack-level battery backup, and redundancy moves from n → n+C → n+2C. Built solutions include automatic medium-voltage failover and automated load-shedding, plus a global firmware update that remediates ~100 UPS units.
AI-written summary · about
chapters (10)
- 0:00Engineering resiliency at scale
- 1:08AWS infrastructure scale
- 2:11Blast radius in power design
- 3:36N plus 2C redundancy
- 5:01Failure testing and self-healing
- 6:27Power infrastructure anomalies
- 8:14Disruptions at scale planning
- 8:35Virginia power event case study
- 10:41Virginia incident lessons
- 11:23Resiliency evolution recap
related talks
35mBehind the Scenes of the Cloud
26mThe tools and techniques behind resilient systems
43mSustaining Everything, Everywhere, All at Once!
14mHyperscale Infrastructure Q&A
27mLive Panel: Building at the Roofline — Planning and Delivering Gigawatt-Scale AI Infrastructure
48mResiliency and Availability Design Patterns for the Cloud
46mCloud-Enabling the Electric Grid with Consequence Driven Approaches
29m"Datacenter Infrastructure Innovation"