Build a Cloud Native Asynchronous Messaging System For Scale With Redis
summary
Shows a concrete pattern: use Redis Streams with XADD/XGROUP/XREAD and XAUTOCLAIM to implement an authoritative or ephemeral event log, deployed on Kubernetes with Prometheus metrics and a Redis exporter. Demonstrates a Flask web API writing events, Postgres as the source of truth, and cooperative consumer groups that scale; offers measured throughput (100–150k ops/core, up to hundreds of shards → tens–150M ops/s) and tradeoffs for durability (AOF sync, multi‑AZ replicas). Demo runs end-to-end and highlights trimming, partitioning, backpressure and monitoring.
AI-written summary · about
chapters (15)
- 0:00Redis and cloud native messaging
- 1:24Traditional service struggles
- 3:28Event broker design patterns
- 5:14Authoritative vs ephemeral logs
- 7:23Redis as shared data API
- 9:10Redis Pub/Sub and Streams
- 12:59Durability trade-offs
- 15:09Redis at scale best practices
- 18:16Redis with CNCF tools
- 19:02Live demo: Redis messaging
- 28:25Async messaging failure modes
- 31:37Q&A: Redis vs Kafka
- 32:41Q&A: Memory for streams
- 34:49Q&A: Durable streams with failures
- 35:53Closing remarks
related talks
59mUsing Redis beyond caching
20mHow to Build a High Performance App Using Cloud Foundry and Redis
39mArchitectural design patterns applied to Distributed Systems
42mIt's like 10,000 streams when what you need is a queue
56mRedis Application Patterns in Rails
51mX-Raying Your Redis App: SDK Tuning with a Performance Engineer’s Eye
43mEvent Streams & Data Rapids: Harnessing the Full Power of Messaging Platforms.
24mWhen Real Time Meets Big Data & Analytics