Chaos Like Everyone's Watching
summary
API management platform built from homegrown microservices and an open‑source API gateway uses chaos engineering tooling rather than only unit tests: it has >15 microservices each with ~93% unit test coverage but still ran 87 experiments over four game days in non‑production using Chaos Toolkit (JSON experiments, Prometheus checks, drivers for Kubernetes/AWS) and evaluated Gremlin but rejected hosted execution. Findings include misconfigured timeouts, broken fallbacks, missing instrumentation, and confirmation that the gateway keeps configs in memory when the DB is killed.
AI-written summary · about
chapters (8)
- 0:00Introduction to API platform
- 3:32Why chaos engineering
- 8:07Chaos engineering workflow
- 10:14Experiments as code
- 13:46Chaos Toolkit overview
- 16:36Game days and findings
- 19:24Runtime resilience tests
- 20:45Key takeaways and tips
related talks
35mChaos Engineering WG Deep Dive
49mChaos Engineering: Withstanding Turbulent Conditions in Production
46mDesigning a System for Chaos
37mBringing order to the chaos!
33mImprove resilience with automated chaos engineering
30mA Retrospective: Five Years Later, Was Chaos Engineering Worth It?
29mBoF Deep Dive: Chaos Engineering
38mChaos Engineering: A Step Towards Resilience