Debugging the Routing Tier
summary
Two real incidents show what to expect and what to collect: one app pair has every GET succeed but the immediate PUT fails because the default Spring HTTP client ignores peer FINs and reuses connections (client timeout is five seconds) while HAProxy keepalive is 0.5s; fixing the client timeout resolves it. A second issue shows many 404s causing heavy logging, lock contention and disk I/O contention on routers; CPU profiles and pprof flame graphs point to logger locks, isolating the router on an empty host removes the problem. Capture routing logs, tcpdump, Wireshark, flamegraphs to prove root causes.
AI-written summary · about
chapters (5)
- 0:00Introduction to debugging
- 1:24What is the routing tier
- 3:38Case study one failure
- 18:30Case study two unresponsive API
- 28:40General debugging tips
related talks
28mMy Favourite Errors
13mHoney, I Broke the Things: Debugging Gray Failures...
09mLightning Talk: Honey, I Broke the Things: Debugging Gray Failures in Production!
24mGremlins Exposed: Shining a Light on Mischievous Systems
21mWhen the Cloud Fails: Debugging the "Undocumented"
25mScaling Up, One Network Bottleneck at a Time
21mThe Cloud is Someone Else's Network: Debugging the Invisible
39mHow to Use Common Investigation Practices in Troubleshooting