Chaos Debugging: Finding and Fixing Microservice Weak Points

on YouTube ↗34m

" summary

Demos show three specific tools and concrete results: Squash attaches gdb/dlv/Java debuggers to live Kubernetes containers and patches memory to fix a calculator bug on the fly; Loop records full request transactions in the sidecar, buffers them, only uploads on error and replays the exact failing inputs into a debugger; Gloo Shot injects faults via any service mesh, watches Prometheus thresholds to abort experiments, and verifies a resilient deploy removes cascading 500s.

AI-written summary · about

" chapters (13)

1. 0:00Chaos debugging intro
2. 3:46Debugging with Squash live
3. 6:46Squash for distributed debugging
4. 10:22Safe production debugging
5. 11:45Tracing trade-offs
6. 13:31Service Mesh Interface problems
7. 16:23Capturing bugs with Loop
8. 19:42Loop’s open-source replay
9. 20:47Chaos as control system
10. 23:18Fault injection with Blue Shot
11. 25:11Blue Shot demo
12. 29:35Service Mesh Interface start
13. 32:27Tools and projects summary

" related talks

• 50mTroubleshooting & Debugging Microservices in KubernetesRay Tsang, Robert Kubis · GOTO Chicago 2019
• 43mDebugging and Troubleshooting MicroservicesBaruch Sadogursky, Ray Tsang · swampUP 2017
• 50mTroubleshooting & Debugging Microservices in KubernetesRay Tsang, Robert Kubis · GOTO Berlin 2018
• 26mDebugging Clojure Services Under StressPaul Lam · Clojure/north 2020
• 42mMicroservice Testing TechniquesDaniel Bryant · Craft Conference 2023
• 41mDebugging an Application with Service MeshAlejandro Pedraza · KCD Guatemala + Profesionales 2021
• 25mDebugging at Scale in Production with kubectl debugShai Almog · Conf42 Cloud Native 2023
• 41mResillient Microservices without the ChaosChristopher S. Meiklejohn · Strange Loop 2022