Bring Your Chaos to Work Day
summary
Runs chaos experiments against staging and production monitoring (Grafana, Loki, Tempo, Thanos, Prometheus, Alertmanager) using Chaos Mesh for kill 25% of pods, OOM and CPU stress, network delay, k6 for synthetic users, and Linkerd for split traffic 50/50 fault injection. Tests cover Alertmanager, Prometheus distributor, Grafana, Thanos and Tempo with four scenarios each. Finds only one P1 alert and mostly Linkerd P3s, so alert levels get tightened and Linkerd timeouts/retries are applied; dashboards combine Linkerd metrics and Loki logs and the event is gamified with assigned roles.
AI-written summary · about
chapters (7)
- 0:00Introduction and motivation
- 3:58Why chaos engineering
- 6:29Bring Your Chaos to Work Day
- 10:01Preparation and tools
- 15:46Running chaos experiments
- 18:56Results and findings
- 26:25Future plans and conclusion
related talks
36mConfidence with Chaos for Your Kubernetes Observability
30mBringing Order to Chaos: Make Your Systems More Resilient with Chaos Engineering
39mMake Cloud Native Chaos Engineering Easier - Deep Dive into Chaos Mesh
37mConfidence with Chaos for your Kubernetes Observability
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
25mDestroy Your System To Make It More Reliable: An Easy Way To Start Chaos Engineering With LitmusChao
40mJourney into the World of Chaos Engineering
32mTesting Kubernetes apps' observability end-to-end