Simplifying Logs, Events and Streams: Kafka + Rails
summary
Uses Kafka as a durable append‑only stream for Heroku router logs, producing JSON from a Sinatra app with an async producer and connection‑pooled clients; consumers run in consumer groups to aggregate metrics into Redis and a separate JRuby replay consumer replays traffic to staging. Library choice is ruby-kafka (Zendesk) or JRuby Kafka, default delivery is at least once, and small clusters handle hundreds of thousands to millions messages per second. Demo is dockerized and the pattern is used in production (Skylight, Heroku event bus).
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:50Friday Hug
- 5:03What is Kafka
- 7:04Skylight
- 8:51Kafka Terminology
- 10:24Kafka Topics
- 12:29Kafka Consumer Groups
- 16:26JRuby Kafka
- 22:18Kafka API
- 24:31Demo app
- 26:33Router logging
- 28:46Kafka producer
- 30:58Heroku addon
- 31:46Kafka CLI plugin
- 32:45Kafka Cluster Inspection
- 33:41Kafka Consumer Inspection
- 35:23Kafka Replay Consumer
- 38:52Kafka Activity Tracking
- 39:55Kafka API Event Bus
- 40:59Conclusion
related talks
33mI Can’t Believe It’s Not A Queue: Using Kafka with Rails
31mEvent Streaming on Rails
35mSo You’ve Got Yourself a Kafka: Event-Powered Rails Services
30mRuby in the Billions
41mKafka as a Platform: The Ecosystem from the Ground Up
50mIntroduction to Apache Kafka
01h 00mIn Depth Kafka Streams
39mKafka as a Platform: The Ecosystem from the Ground Up