Case Study — How We Block Production
summary
A case study of a production outage in an ESB-based system using RabbitMQ and asynchronous microservices where a runaway Ruby container repeatedly requeued messages, saturating the network and causing RabbitMQ cache poisoning. The incident showed an anomalous node sending ~300 Mbps versus normal ~30 Mbps, leading to prefetch and memory exhaustion and stuck old messages; recovery (~1h20m) involved killing the container and adding Docker memory limits, plus new metrics and alarms for frequent restarts and oversized packets.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:26Project overview
- 5:50Production issue detected
- 8:11Incident checklist
- 10:13Load balancer problem
- 15:31Root cause analysis
- 18:51Solution and recovery
- 20:38Monitoring improvements
- 22:30Q&A
related talks
21mWhy does RabbitMQ stop and what can be done about it
22mLessons Learned from Scaling RabbitMQ in a High-Growth (...)
18mHow to Break RabbitMQ
39m101 ways to break your RabbitMQ
32mRabbitMQ in Real Life - Tips and Tricks
48mBEAM all the things
19mBuilding a Distributed Data Ingestion System with RabbitMQ
29mPost Mortems for 4 Years of Remote Execution