Scaling RabbitMQ
summary
SoundCloud moves from a single RabbitMQ broker to clustered, multi-broker topologies, using AWS autoscaling for workers and a load balancer for publishes while consumers connect to all backends. The setup handles 20–30k messages/sec at peak, recovers tens of thousands backlog within hours, and isolates activity feeds into a separate cluster to avoid global failure. Engineering adds heartbeats, a homegrown AMQP client for Go, a model broadcast pattern for state changes, and works toward service discovery, semantics events, and commodity-node scaling.
AI-written summary · about
chapters (11)
- 0:00Intro
- 7:50Transcoding
- 10:53RabbitMQ
- 11:55Autoscaling
- 14:08Mapping to RabbitMQ
- 16:44Scaling Issues
- 18:39Model Broadcast
- 24:38The Solution
- 26:15Changes
- 28:48Benefits
- 34:38Abstract AMQP
related talks
24mAdvanced RabbitMQ Strategies: A case study
22mLessons Learned from Scaling RabbitMQ in a High-Growth (...)
01h 05mScaling Applications with RabbitMQ
32mBuilding distributed system for retail with RabbitMQ
19mBuilding a Distributed Data Ingestion System with RabbitMQ
40mBuilding Scalable Resilient IoT Messaging
39mReal World RabbitMQ: Exploring Practical Applications
30mA love letter to messaging: Celebrating RabbitMQ's journey