Event Streaming on Rails

on YouTube ↗31m

" summary

Large Rails monolith (about 43,000 Ruby files and thousands of routes) migrates to Kafka-based event streaming using the Ruby Kafka gem, Kafka Connect and CDC to avoid dual-write and enable region-to-region replication and replays (retention ~one week). A transactional outbox pattern ensures transactional consistency, Kafka multicasts allow independent consumers, and Kafka Connect / hosted MSK or Confluent handle ops; the pipeline is built and works but requires careful idempotency and schema management.

AI-written summary · about

" chapters (14)

1. 0:00Welcome
2. 1:05Procore’s Rails scaling
3. 3:10Event streams and Kafka
4. 4:32Kafka from Ruby
5. 7:43Why Kafka for Rails
6. 11:30Service communication in Rails
7. 13:58Multicast with event streams
8. 16:24Kafka Connect and data stores
9. 18:30Event sourcing and CQRS
10. 22:00Replication strategies
11. 23:22Kafka in Rails apps
12. 25:29Dual writes and CDC
13. 28:16Consistent producers
14. 28:58Summary and future

" related talks

• 27mEvent Streaming Patterns for Ruby ServicesBrad Urani · RubyConf TH 2023
• 42mSimplifying Logs, Events and Streams: Kafka + RailsTerence Lee · EuRuKo 2016
• 35mSo You’ve Got Yourself a Kafka: Event-Powered Rails ServicesStella Cotton · RailsConf 2018
• 11mSpinning up micro-services using Ruby/KafkaAnkita Gupta · Red Dot Ruby Conference 2017
• 33mI Can’t Believe It’s Not A Queue: Using Kafka with RailsTerence Lee · RailsConf 2016
• 30mEvent Streaming and the Future of ApplicationsBen Stopford · JAX 2019
• 11mSpinning up micro-services using Ruby/KafkaAnkita Gupta · RedDotRubyConf 2017
• 21mHow Apache Kafka® Generates Value In Retail With ConfluentAnna Semjen · Big Data LDN 2022