Breaking Storage Barriers: How RabbitMQ Streams Scale Beyond Local Disk
summary
Claim: RabbitMQ Streams uses a pluggable file-abstraction layer so only the active tail needs replication, shrinking node join time from ~2 hours for 6 TB to seconds by fetching ~50 MB. Implementation keeps the write path unchanged, adds a background archiver to S3 and a S3 plugin that implements manifest and reader behaviors, and serves archived reads with prefetching to preserve throughput at an expected ~50 ms startup latency. Code and PRs are open source.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:10What are RabbitMQ Streams
- 2:55Replication and storage challenges
- 4:43ABBA ticketing disaster
- 7:50Redefining storage scalability
- 9:56Designing remote storage plugin
- 12:05Writes remain unchanged
- 13:06Local vs remote storage reads
- 14:31Scaling new nodes in seconds
- 15:17Solving ABBA ticketing crisis
- 16:01Open source S3 plugin future
- 17:27Call to action
- 18:08Q&A
related talks
35mVaronis journey deploying RabbitMQ
41mPushing RabbitMQ to the limit
16mTo queue or not to queue, that is the question.
31mServing millions of clients with Native MQTT and MQTT 5.0
41mDesign (Failures) Case Study
25mMixing data streams at scale
30mSupport Tiered Storage in Strimzi Operated Kafka
40mDistribution, Scale and Flexibility with ZeroMQ