Automatic Traffic Scheduling for Internet Connectivity Failures
summary
Baidu’s system detects 3–5 PoP outages and 5–10 backbone breaks weekly and classifies failures as PoP, backbone, or local ISP by aggregating probe failures across province/area/datacenter. It runs HTTP probes every 10s from a P2P CDN with 3000+ probes, marks a link abnormal when abnormal results exceed a 20% threshold, and raises alarms within a minute. Traffic scheduling uses DNS/IP replacement plus a GGC controller that avoids overloading 100 GB links and rebalance services (e.g., shifting 500 QPS when incoming load rises from 2500 to 3000 QPS), reaching recovery <10 minutes.
AI-written summary · about
chapters (15)
- 0:00Start
- 0:08About me
- 1:19Two ways to handle the Internet connectivity failures
- 2:46Baidu network architecture
- 4:07Data center PoP(point of presence) network failures
- 4:55ISP backbone network failures
- 5:48Local ISP network failures
- 7:29The impact of Internet connectivity failures
- 9:42How to detect these failures
- 11:45The agent network situation is not stable
- 14:29Diagnose the fault location
- 16:21Automatic traffic scheduling GTC the Global Traffic Control system
- 17:34The result of traffic scheduling
- 19:05The limit of service capacity
- 20:12Capacity of backend service
related talks
21mAAsclepius: Monitoring, Diagnosing, and Detouring at the Internet Peering Edge
24mHarp: Improving VPC Network Availability via Efficient Failure Detection and Rerouting...
23mNetBouncer: Active Device and Link Failure Localization in Data Center Networks
17mReasoning about Network Traffic Load Property at Production Scale
24mBlink: Fast Connectivity Recovery Entirely in the Data Plane
15mCost-effective and Reliable Global Internet Peering with Programmable Switches
29mUsing the Internet as Your Load-Balancer
04mLoad balancing through the years