Kafka as a Message Queue: can you do it, and should you do it?
summary
Demonstrates a Kafka-based message queue called kmq that uses two topics (queue + markers), writes start/end markers, and runs a redelivery tracker with an in‑memory priority queue to implement selective acknowledgments and a visibility timeout 15s. Claims at‑least‑once delivery with possible duplicates, built with plain Kafka consumer/producer APIs. Measured throughput: plain Kafka ~16k msgs/s, kmq slightly lower; 95th‑percentile latency ~50ms for plain Kafka, spiking to ~130ms under heavy load.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 2:24How Kafka works
- 8:49Why use Kafka
- 11:02Amazon SQS
- 13:03Kafka Message Queue
- 14:31How does it work
- 16:10Position of the message
- 19:43Delivery Tracker
- 21:41In practice
- 23:02Kafka setup
- 26:56Kafka client
- 32:35Troubleshooting
- 36:21Performance
- 40:08Latency
- 41:16Dropping
- 43:55Summary
- 45:00Links
- 45:51Questions
related talks
43mUnifying streaming and message queue with Apache Kafka
38mKafka Message Broker Under High Load
53mDemystifying Kafka
44mFive Years of the Good and the Not-So-Good with Kafka
42mA Kafka Client’s Request: There and Back Again
35mKafka Internals I Wish I Knew Sooner
05mIntroduction to Kafka Internals
33mKafka Microservices with Kafka plus Akka and Kafka Streams