Introducing Stream Support in RabbitMQ
summary
RabbitMQ 3.9 adds a replicated, persistent append-only stream type with non-destructive reads and server-side offsets, plus a new binary stream protocol and plugin. Benchmarks show >1M msg/s on a 3-node cluster (scaling near linearly to ~5M with more streams; special batching mode reaches tens of millions in synthetic tests). Features include publish-confirm durability, broker deduplication, retention truncation, TLS and sendfile-backed disk I/O, a Java reference client and a Go work-in-progress client for immediate experimentation.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:29Streams: a new type of data structure in RabbitMQ
- 2:02Streams Strong Points
- 6:06Fan-out with traditional que
- 7:16The Log Abstraction
- 9:07Consuming from the end of a stream
- 9:44Consuming from a point in the stream
- 10:41A consumer can read the stream several times
- 15:25Consuming from a Stream with any AMQP 0.9.1 Client
- 17:01Streams on AMQP
- 18:45Stream Protocol
- 19:44Streams are also accessible through a new protocol
- 20:36Tooling
- 22:21Stream Interoperability
- 23:20Traditional Routing with AMQP
- 24:08Add Stream for Analytics
- 27:25Performance (sub-entry batching)
- 29:53Guarantees
- 31:50Other features
- 34:09Java Client: Stream Management
- 34:53Java Client: Resource Allocation
- 35:07Java Client: Producer declaration
- 36:54Java Client: Configuration based
- 37:43Java Client: Connections are re-used
- 38:53Roadmap
related talks
42mWhat's New in RabbitMQ Streams?
15mUpdate from RabbitMQ Core Team
37mLet's build our own message streaming platform
26mBreaking Magical Barriers
57mGame of Streams: How to Tame and Get the Most from Your Messaging Platforms
37mImplementing Raft in RabbitMQ
59mGame of Streams: How to Tame & Get the Most from your Messaging Platforms
28mKeynote: The State of NATS as a Reactive Substrate