Building a Distributed Data Ingestion System with RabbitMQ
summary
Claims a working pipeline that replicates geo‑located producers into a central cluster using RabbitMQ Federation to avoid building custom replication, with policies like expires and max_hops to prevent unbounded local queues. Demonstrates multi‑protocol producers (AMQP, MQTT, STOMP), fsync replication semantics (confirm vs no‑ack) and a sharding approach using the Consistent Hash Exchange to spread load across queues. Notes scale: 1 billion messages/day is only ~11,000/sec and describes production usage and operational knobs that make it actually work.
AI-written summary · about
chapters (5)
- 0:00Introduction to RabbitMQ
- 2:42Messaging basics
- 5:09Routing algorithms
- 9:05Replication challenges
- 13:00Federation and scaling
related talks
48mBuilding a Distributed Data Ingestion System with RabbitMQ
01h 19mBuilding a Distributed Data Collection System with RabbitMQ
32mBuilding distributed system for retail with RabbitMQ
40mBuilding Scalable Resilient IoT Messaging
22mLessons Learned from Scaling RabbitMQ in a High-Growth (...)
40mScaling RabbitMQ
48mA Beginner's Guide to Inter-Service Messaging
24mAdvanced RabbitMQ Strategies: A case study