Breaking Magical Barriers
summary
Claim: a single replicated RabbitMQ queue reaches 1,000,000 messages/sec (up from ~25k) by changing the replication and I/O plumbing rather than the VM or language. Achieved with new binary protocol, dedicated TCP links using sendfile-backed TCP to move disk straight to socket, sequential disk writes and batching, and a simple durable replicated log called streams that enable time-travel message replay. Classic mirrored queues use chain replication and incur 5x replication overhead; quorum queues use Raft and cut that to 2x, but streams eliminate those costs and shift the bottleneck to CPU frequency.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 2:08Bugatti’s 300 mph vs RabbitMQ’s 1M msg/s
- 4:43Erlang runtime powers RabbitMQ
- 7:55Disks as RabbitMQ’s tires
- 13:18Aerodynamics in RabbitMQ clusters
- 15:24Classic mirrored queues slow
- 17:30Quorum queues limits
- 18:52Streams 36x speed boost
- 20:40What are RabbitMQ streams
- 22:24Streams’ speed and future
- 23:10Credits and contributors
- 23:53RabbitMQ future with Erlang JIT
- 25:17Modern benchmarking platforms
- 25:40Closing thoughts
related talks
50mPanel discussion: The new RabbitMQ Streams
51mKeynote: Quorum Queues: A Retrospective
44mKeynote: Erlang Powered Rabbits
53mRabbitMQ Performance Improvements
36mDRb vs. RabbitMQ Showdown
37mLet's build our own message streaming platform
42mWhat's New in RabbitMQ Streams?
50mIntroducing Stream Support in RabbitMQ