Testing for DR Failover Testing
summary
Zendesk Chat runs its backend in one datacenter and failover tests initially fail often (first exercise needs three tries, later one attempt succeeds). The solution runs end-to-end browser tests against the DR standby using test accounts replicated from production, allows writes to DR for those accounts, blocks risky changes by avoiding auto-increment tables, and enforces safety with a DB user with restricted permissions. This discovers real issues during dry runs and raises confidence that a real failover will work.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 8:35DR Failover Testing Challenges
- 10:27DR Failover Tradeoff
- 13:36Data Source
- 14:59Different Process
- 20:22Should we allow them
- 20:47Auto increment columns
- 23:22Magic user
- 24:15India
- 26:31Credits
- 26:47Summary
- 27:35Outro
related talks
46mChaos Experiments - Datacenter Stress Testing
28mHow Not-To Test in Production: Learnings from DoorDash
22mConfused Tester in Chaotic World
25mChaos Engineering for Cloud native Apps
01h 08mPerformance testing in the cloud
48mTesting The Cloud
44mChaos Engineering for SQL Server
41mTesting on Production, deep backend edition