Varonis journey deploying RabbitMQ
summary
Varonis deploys thousands of single-node RabbitMQ instances to over 7,000 customers and solves slow networks by sending compressed batches of up to 300 messages or every 100 ms via a custom packing exchange, cutting network use about 10x. Persistent bursts are offloaded by a storage exchange that writes to disk and replays when consumers recover, keeping Rabbit startup to seconds instead of hours. Additional plugins: a capping exchange for coordinated rejects and a watchdog for self-healing.
AI-written summary · about
chapters (12)
- 0:0002:55 - Introduction
- 2:5604:56 - We are not the Common Use Case
- 4:5705:46 - Who we are
- 5:4707:26 - Goal #1 - Supporting High Throughput & Less Network
- 7:2710:41 - Packing Exchange More Throughput and Less Network Usage
- 10:4214:39 - Goal #2 - Resiliency to Slow/ Stopped Consumers
- 14:4020:04 - Storage Exchange - Resiliency to Consumer Down-Time
- 20:0523:59 - Storage Exchange - Why not Store in RabbitMQ?
- 24:0026:35 - Resiliency to Consumer Down-Time Method #2
- 26:3628:08 - Goal #3 - Guarding RabbitMQ
- 28:0929:58 - Watchdog Pluggin, Monitoring RabbitMQ
- 29:5934:39 - QnA
related talks
19mBreaking Storage Barriers: How RabbitMQ Streams Scale Beyond Local Disk
46mFollow the RabbitMQ
41mPushing RabbitMQ to the limit
45mDecentralizing SaaS Monitoring Using Open-Source Tools and Standards
31mServing millions of clients with Native MQTT and MQTT 5.0
15mHow Our SREs Safeguard Nanosecond Performance—at Scale—in an Environment Built to Fail
41mDevOps' Swiss Army knife for the BEAM
43mRealtime performance at scale (...)