Effective Data Synchronization between Rails Microservices

on YouTube ↗38m

" summary

Claim: a Kafka-based message bus lets data engineers produce Avro-validated operations and Rails apps consume them through importers that run real domain logic, with per-app topics, batching, permitted-attribute checks, and backpressure controls. Producers often run Python jobs that write Avro-validated operations into batches; consumers map model+type to importers and return operation results. The system yields 7.7 billion data updates since April 2019 and proves Kafka-based message bus and per-app topics scale and preserve Rails callbacks.

AI-written summary · about

" chapters (14)

1. 0:00Introduction
2. 1:58Agenda
3. 2:34Company Background
4. 5:00Effective Data Synchronization
5. 5:46Data Updates
6. 9:15Data Needs
7. 9:58Example
8. 12:55The Pinky Promise
9. 19:09The Growth
10. 24:44Operations
11. 26:29Batching
12. 28:09Data Producers
13. 31:39App Consumers
14. 35:39Summary

" related talks

• 35mSo You’ve Got Yourself a Kafka: Event-Powered Rails ServicesStella Cotton · RailsConf 2018
• 11mSpinning up micro-services using Ruby/KafkaAnkita Gupta · Red Dot Ruby Conference 2017
• 31mEvent Streaming on RailsBrad Urani · RailsConf 2022
• 27mEvent Streaming Patterns for Ruby ServicesBrad Urani · RubyConf TH 2023
• 11mSpinning up micro-services using Ruby/KafkaAnkita Gupta · RedDotRubyConf 2017
• 33mI Can’t Believe It’s Not A Queue: Using Kafka with RailsTerence Lee · RailsConf 2016
• 30mSome Funny Things Happened on The Way to A Service EcosystemChris Hoffman · RailsConf 2018
• 42mSimplifying Logs, Events and Streams: Kafka + RailsTerence Lee · EuRuKo 2016