The Mushroom Cloud Effect or What Happens When Containers Fail?
summary
Dynatrace instrumentation places an agent on each host, injects a JVM agent and a JavaScript snippet into HTML to correlate user and backend traces, then analyzes service dependencies to find root causes. A production case shows shared log partition exhaustion plus leftover container files fills root and Docker storage, causing health-check restart loops that ripple into a mushroom cloud of failures. A three-day stress test exercises 13,000 containers across 3,400 services and 820 billion dependency edges and pins the outage to two hosts with TCP problems.
AI-written summary · about
chapters (26)
- 0:00Intro
- 1:20About Dynatrace
- 4:40How about you
- 6:45Introduction
- 7:55What is the Mushroom Cloud Effect
- 10:40How people build the system
- 15:35Microservice environment
- 17:50Request resolution
- 19:50Mushroom cloud effect
- 21:23What was the problem
- 22:33What happened
- 28:46What did they do
- 30:10The Mushroom Cloud
- 31:28What is the point
- 32:31Large container moments
- 33:21Demo environment
- 33:38Demo dashboard
- 34:51Demo applications
- 35:27Service flow
- 35:57MongoDB
- 36:46Microservices
- 37:22Dependencies
- 38:40Response Time
- 39:25Mushroom Cloud
- 40:30Application Monitoring
- 45:02Go Support
related talks
30mWhat Happens When Containers Fail
38mCreating chaos in containers
14mLeveraging Distributed Tracing in Kubernetes & Containerized Environments
37mCreating chaos in containers
22mAll Clouds Need a Sprinkle of Chaos
52mPerfecting reliable code delivery for the cloud
25mSpend more time innovating, less time troubleshooting
49mIn Silicon Valley, Engineers Deploy-and-Tell: Stories About Dockerizing an Application