Live Traffic Load-Testing-Measuring and Validating Capacity at Facebook
summary
Facebook runs 80,000 tests per day across ~250 services and consumes one-third of capacity to measure maximum servable load; service tests ramp traffic until median CPU hits stop at 80% CPU and use a modular pipeline of checker controller actuator so new traffic loaders or spike patterns plug in without rebuilding the system. Region tests coordinate with traffic management to hit an ideal load safely, surface load‑balancing hotspots, and reveal surprising results like some services failing well before expected peaks while others report >100% scale, guiding capacity and regression decisions.
AI-written summary · about
chapters (11)
- 0:00Live traffic load testing intro
- 1:31Capacity validation importance
- 4:18Evolving workloads challenges
- 6:25Live production traffic benefits
- 7:50Service-level load testing
- 10:39Scaling diverse services
- 13:5680,000 daily tests insights
- 14:19Region-level ecosystem validation
- 15:22Safety in regional tests
- 17:48Years of testing findings
- 19:57Future load testing directions
related talks
46mLessons learned from monitoring a large production network
16mEvolvable Network Telemetry at Facebook
41mDrowning on Metrics or why Jack could have fit
28mReliability Lessons from Running High Traffic Services for More Than 10 Years
54mEverything Everywhere at Once: Tracing under High Load with 10k Backends
40mKeynote 'How to explain DevOps to your mom and how it works at Facebook, PayPal and others!'
28mProduction Testing Through Monitoring
24mMeasuring the Reliability of Live Video Infrastructure