Surviving Black Friday - A resilience engineering tale
summary
Retail site resilience focuses on concrete fixes: an in-house Hydra latency injector simulates 5–20s delays and errors across services, tests run repeatedly against a dependency-by-flow matrix to find hidden coupling, timeouts are set from percentile targets, backpressure limits threads, carts are persisted to localStorage during outages, features use gradual rollout (10% → 100%) and feature toggles, circuit breakers auto-open/close to stop cascading failures, and Graphite/Grafana plus anomaly detectors visualize impact so fixes are measurable and verified under load.
AI-written summary · about
chapters (9)
- 0:00The challenge of Black Friday
- 3:00Understanding system failure
- 8:23Step 1: Know your priorities
- 11:14Step 2: Simulate and test
- 15:50Step 3: Fail fast
- 18:38Step 4: Fail gracefully
- 23:22Step 5: Control features
- 25:04Step 6: Automate protection
- 28:11Step 7: Measure everything
related talks
29mBlack Friday - Lessons in Resiliency and Incident Response at Shopify
42mHow to stay alive even when others go down:...
30mIncident Groundhog Day
33mFrontend Resilience
16mEngineering Failure-Resilient Systems
34mBuilding High Resiliency into Digital Solutions
22mResilience Is Not Enough
28mReliability Lessons from Running High Traffic Services for More Than 10 Years