Chaos-Driven Development: TDD for Distributed Systems
summary
Chaos experiments are applied as TDD to a log aggregation HA design on public cloud, integrating alarms, metrics and failover logic before running faults. Actual injected failures reveal missing steady‑state metrics, lost processor logs, a too‑eager failover, file overwrites from timestamp names, and node failure caused 3–4 minute alarm delay that defaults hide. The team codifies expectations as automated checks, reconfigures storage redundancy and naming, and measures real limits against 300 billion ticks per day to set realistic SLOs.
AI-written summary · about
chapters (9)
- 0:00Chaos-Driven Development: TDD for Distributed Systems
- 4:17Resilience-First Development
- 5:23Log Aggregation Architecture
- 8:32"Unit Tested" Failover Logic
- 10:14Confronting Our Design Choices • Are we too hasty to failover? Node Failure and Pod Failure experiments
- 12:48SLOS & Application Limitations
- 13:25SLOS & Platform Limitations
- 14:01Observability: A Force Multiplier!
- 15:09Setting Up Future Success
related talks
46mDesigning a System for Chaos
57mChaos Engineering Primer
33mImprove resilience with automated chaos engineering
42mChaos Engineering: Why Breaking Things Should be Practiced
18mFostering Chaos Engineering Culture
37mBringing order to the chaos!
39mTry Catch Blocks for your Distributed System
55mChaos Engineering for Cloud Foundry Platform