Planning for Overload
summary
Argues that overload must be planned for not treated with bigger limits alone, and shows concrete tools and results: use po_box / batch_io at Heroku to buffer and batch IO into 4KB writes, set a 40,000-message buffer and accept controlled loss, which stops log-induced crashes (from ~monthly to effectively none). Describes explicit backpressure and load-shedding options — safety valve, SASL overload, memsup/cpusup/disk_stop — and gives sample rates (10,000 req/s × ~50 log events) to justify sampling, queuing, or dropping.
AI-written summary · about
chapters (4)
- 0:00Start
- 5:43Overload must be planned for it defines how you die it defines premature optimization it defines your margin of error it defines your API it defines your engineering
- 19:14random drop when some loss is acceptable (sample size!) can be made adaptive works even better producer-side
- 23:42implementations lager does it on OTP errors for cascading failures
related talks
30mAbove the Clouds: Investigating Systems from 32,000 ft.
32mResiliency and Availability Design Patterns for the Cloud
49mManaging and Monitoring Marklogic for Reliability At Scale: A Customer Use Case
35mHigh Throughput with Low Resource Usage: A Logging Journey
43mSLO Burn—Reducing Alert Fatigue and Maintenance Cost in Systems of Any Size
41mA Theory and Practice of Alerting with Service Level Objectives
29mCapacity Planning with the Universal Scalability Law
28mPrepare Your System To Scale OR Why Auto Scaling Is Not Enough