Preventing cascading failures in a global network
summary
Global CDN capacity was about 2–3 TB/s two years ago and now grows fast; containment is layered. Per‑box isolation uses control groups to prevent logging or ingestion processes from stealing CPU or memory from the Sailfish web server. At pup level a pup handles ~1,000,000 req/s so when traffic hits 1.1M the plan is to fail fast and return 503/429 rather than degrade everything (this rate‑limit has not yet been needed). Globally traffic is split by DNS into regional domains so an APEC outage won’t affect NA/EU.
AI-written summary · about
chapters (6)
- 0:00O'REILLY Velocity
- 0:16Cascading failure at scale(s)
- 1:10What is a cascading failure?
- 2:38Containment on a single system
- 3:14Containment in a POP
- 3:57Containment globally
related talks
44mTechniques Netflix Uses to Weather Significant Demand Shifts
44mAvailability, Latency and Cost: Withstanding Outages
25mWe're Still Down: A Metastable Failure Tale
30mNoisy Neighbors, through Networking
20mBetter Resilience Through Self Tuning
34mDistributed Systems - What Can Go Wrong and How to Get It Right
29mSex, Lies & Videotapes: The Truth Behind Your CDN Logs
18mTGW: Operating an Efficient and Resilient Cloud Gateway at Scale