Hijacking Service Discovery to Simulate Dependency Degradation
summary
Shopify runs 170M RPM average and built tooling to simulate degraded dependencies without changing service code: a fault-injection proxy Toxyroxy, a Kubernetes operator game-day-body that injects hostAliases and spins up proxies, and a Zookeeper-aware proxy that rewrites discovery responses. It injects real toxics (examples: 1,000 ms latency, bandwidth limits), routes traffic via coreDNS or by mutating ZK nodes, and successfully ran experiments—e.g., isolating an entire US‑central database slice with a single config.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:19Shopify resiliency team
- 2:23Game days purpose
- 4:55Fault injection proxies
- 6:59Code change limits
- 7:43Game Day Body operator
- 8:25Local DNS hijacking
- 9:51Global DNS challenges
- 10:34Zookeeper hijacking
- 13:03Hijacking discovery layers
- 14:27Zookeeper proxy targeting
- 15:11Summary and call to action
related talks
26mResiliency Testing with Toxiproxy
28mBuilding and testing resilient applications
46mDesigning a System for Chaos
39mTesting Level Hard: Distributed Apps
23mTrustworthy Graceful Degradation: Fault Tolerance across Service Boundaries
28mHow Not-To Test in Production: Learnings from DoorDash
41mData Driven and Containerized Testing of Multi Service Networked Systems
06mIt's Just a Jump to the Left...