Chaos Experiments - Datacenter Stress Testing
summary
A financial-services SRE program automates full single-data-center failover via a CI/CD pipeline with pre-checks, an SLO management tool that enables SLO burn-rate auto-backoff, and an internal status page plus a downloadable containerized load-test starter kit. Tests scale from single JVMs to half-capacity evacuations, catch defects (a POC shows 90–95% of requests survive a degraded tier), handle ~2–3k weekly changes, reduce manual toil, and measurably prevent outages while shifting culture over 15 years.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:05Datacenter migration history
- 8:44JVM to full testing
- 13:21Automating chaos with SLOs
- 18:58Cultural hurdles in chaos
- 25:42Handling large findings
- 30:37Team engagement strategies
- 35:52Overcoming fear uncertainty and doubt in leadership
- 42:35Real-world impact and roadmap
related talks
25mBuilding a highly scalable system to assure Business Resilience
57mChaos Engineering Primer
39mResilience & Chaos Engineering in the Cloud
01h 08mPerformance testing in the cloud
08mChaos Engineering at Citi Bank
29mImprove Resilience with automated chaos engineering
35mImprove resilience with automated chaos engineering
34mStress-Test tools in Azure to optimize your reliability