Hearts of Darkness: A Spring DevOps Apocalypse
summary
Shows real failures and fixes: blocking Tomcat request threads exhaust the default 200 Tomcat threads, letting liveness probes time out (2s) and pods get killed, fixed by running actuator probes on a separate connector/port in Kubernetes; circuit breakers (Hystrix → Resilience4j) get stuck in half‑open when ignored 404 exceptions prevent success transitions; Micrometer metrics to Datadog collapse when no instance tag causing a 1/8 metric drop after an upgrade; OOM traced to syslog4j’s unbounded linked list (about 123k linked-list nodes), fixed by bounded buffering then SQS.
AI-written summary · about
chapters (23)
- 0:00Intro
- 1:32Motivation
- 2:57Spring Boot & Health Checks
- 4:23Luck Of The Draw
- 5:50Die Trying
- 9:02Bulkheads Prevent Floods
- 11:35To Each Their Own (Thread Pool)
- 14:44Built-in With Boot 2.6
- 18:32Tomcat Metrics
- 20:03The Killing Fields
- 21:47Custom Status in Health Indicator
- 23:12The Long And Windy Road
- 23:58What Happened?
- 26:34Tripping Out
- 30:26Experiencing Success
- 32:47Other Lessons Learned
- 36:23The Missing Metrics
- 38:17Analysis Paralysis
- 41:20Medic!
- 41:51Educated Guessing
- 42:21Rubber Ducking
- 45:15Taking A Dump
- 49:56Conclusion
related talks
30mThe last in the line: a talk about DevOps and Disasters
41mAfterlife Tales
39m30 minutes of Failure
34mReading 100+ Kubernetes Post-Mortems - lessons learned
51mI Can See Clearly Now: Observability of JVM & Spring Boot 2-3-4 apps
55mDevOps @Scale (Greek Tragedy in 3 Acts)
47mSpring Boot Observability in Practice: Actuator, Micrometer & OpenTelemetry
31mEscaping operations hell