The Cloud is Someone Else's Network: Debugging the Invisible
summary
A real incident shows users seeing 504 vs 200 mismatch while application logs report 200; tracing and load‑balancer logs reveal many requests at ~31–32s and a 30s LB timeout. The fix is concrete: increase LB timeout to 45s and align timeouts across API gateway, Ingress and app, add distributed tracing, keep request‑flow diagrams and runbooks current, and use observability plus session replays like Sentry LogRocket FullStory to catch disappearing requests.
AI-written summary · about
chapters (11)
- 0:00Cloud mystery introduction
- 1:14Disappearing requests
- 2:37Debugging like Sherlock Holmes
- 3:44Cloud detective tools
- 5:08Forensic cloud tools
- 7:15Cloud crime scene mapping
- 9:00Cloud network suspects
- 12:08Cloud forensic evidence
- 16:40Phantom request revisited
- 19:50Cloud mystery lessons
- 20:31Cloud detective advice
related talks
21mWhen the Cloud Fails: Debugging the "Undocumented"
31mParting the Clouds: Tracing Cloud Foundry for Improved Performance...
24mWhen Connections are Magic: Understanding Performance in Serverless
29mMaking Sense of Chaos: Operational Visibility in Distributed Systems
36mObserving all the Serverless Things
31mDebugging the Routing Tier
40mGood, Bad, and Ugly - the Art of Load Balancing at Scale
39mConnecting the dots: observability in large e-commerce environments