The 900-Second Ghost: Debugging Half-Open TCP
summary
Production MQTT fleet (2M+ concurrent, 15M daily users, 3.2M RPM) hits a 15‑minute outage caused by half‑open TCP connections; packet captures with tcpdump/Wireshark show FIN packets dropped by cloud load balancer, TCP retransmits keep the socket alive until tcp_retries2=15 — totaling 900 seconds — and an MQTT keepalive set to 15s never fires because the Paho client’s write path blocks. A patch to the library and coordinated changes with the cloud provider fix FIN delivery without accepting ungraceful RSTs.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:05MQTT platform scale
- 3:17MQTT for real-time monitoring
- 5:0815-minute downtime incident
- 6:32Impact of half-open TCP
- 7:15Half-open connections explained
- 10:07L4 vs L7 keep-alive
- 11:51MQTT keep-alive detection
- 13:38Incident questions start
- 14:40Replicating in staging
- 16:24TCP packet capture analysis
- 18:34900-second retransmission mystery
- 20:24Keep-alive timeout failure
- 21:30Failure chain blocking keep-alive
- 22:36Fixing MQTT library
- 22:57FIN packets dropped by LB
- 24:00Graceful vs ungraceful termination
- 25:26Cloud provider collaboration
- 26:10Learnings from debugging
- 27:11Q&A
related talks
21mWhen the Cloud Fails: Debugging the "Undocumented"
20mNetwork Monitor: A Tale of ACKnowledging an Observability Gap
41mWar stories from the CloudAMQP support
13mConnection Storms: Detecting and Defusing Outages Before Valkey Crashes
39m30 minutes of Failure
30mNoisy Neighbors, through Networking
22m007: Democratically Finding the Cause of Packet Drops
31mDebugging the Routing Tier