connect() - why you so slow?!
summary
Opening ~50k connections to one origin makes connect() latency bimodal and wildly slow: experiments and flamegraphs show the inet_hash_connect loop plus parity (even/odd) port walking causes massive looping and lock contention, yielding ~279 connects/sec and exhaustion taking minutes. Cloudflare uses the work stress tool, flamegraphs, BPF tracing and a Prometheus eBPF exporter to profile. User-space fixes include bind-before-connect with random picks and the IP_LOCAL_PORT_RANGE socket option with 1K port windows and random offsets, dropping exhaustion from ~3 minutes to ~9 seconds.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:43Cloudflare connection challenges
- 7:27inet_hash_connect algorithm
- 12:28Looping problem analysis
- 16:00User space solutions
- 20:35Local port range optimization
- 21:39Kernel policy discussion
- 22:24Q&A session
related talks
36mWhat Is Happening with My Connections? Why Are They Closed?
05mJump-starting back-office connections
30mNoisy Neighbors, through Networking
30mOn Sockets and System Calls: Minimizing Context Switches for the Socket API
32mActive Fault Finding in Networks
31mLayer Four and Three Quarters: Fantastic Quirks and Where to Find Them
30mThe Unconspicuous Role of Conntrack in Kubernetes Networking
30mLessons From Defending The Indefensible