Noisy Neighbors, through Networking
summary
Reddit-scale clusters (500k–1M requests/sec) show concrete failure modes: one monitoring path generates 140,000 UDP packets per second from four senders, overloading softirq/CPU and causing 100ms+ latency; pinned CPUs on 96-CPU (48-core) nodes block network processing; connection tracking grows into millions until alerts at ~400,000 sockets. Remediations that work: delete statsd/Telegraph sends, enable receive packet steering, tune conntrack and HPAs, and test eBPF halves iptables CPU which cuts kernel CPU cost and stabilizes SLOs.
AI-written summary · about
chapters (9)
- 0:00Noisy neighbors in networking
- 1:20Reddit’s scale and incidents
- 2:44Telegraph case and UDP overload
- 8:03Session service CPU issues
- 17:48Connection tracking limits with nftables
- 21:25Solutions for tracking and CPU load
- 25:44Fixing Telegraph and CPU accounting
- 26:53Receive packet steering improvements
- 29:46Collaborative efforts and closing
related talks
25mScaling Up, One Network Bottleneck at a Time
32mActive Fault Finding in Networks
35mAll Your Queues Are Belong to Us: The Hunt for a Network Bug...
20mNetwork Monitor: A Tale of ACKnowledging an Observability Gap
30mLessons From Defending The Indefensible
27mDebugging Linux Issues with eBPF
28mPerformance optimization of Neutron server and Agent for large-scale scenario
36mThe black art of network performances