Event Streaming Patterns for Ruby Services
summary
Procore runs a 43,000‑file Rails monolith with 1,123 tables, 4,000 API endpoints and ~300 engineers and solves cross‑service, multi‑region consistency by putting changes into Apache Kafka (MSK → Confluent Cloud), using karafka clients, one‑week retention and replay. Dual‑write issues prompt either Debezium CDC via Kafka Connect or a transactional outbox pattern; consumers must be idempotent because Kafka gives at‑least‑once delivery and guaranteed ordering. Topics auto‑back up to S3 for a hoped‑for replay, and use cases include search (Elasticsearch), reporting (Snowflake), materialized views and cross‑region replication.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:47Procore’s streaming platform
- 5:18Service communication failures
- 7:04Apache Kafka introduction
- 10:37Kafka features and benefits
- 12:24Consistent Kafka pipelines
- 15:53Dual write problems
- 18:43Transactional outbox pattern
- 20:46Kafka consumption and idempotency
- 22:30Kafka use cases at Procore
- 26:22Summary
related talks
31mEvent Streaming on Rails
30mEvent Streaming and the Future of Applications
11mSpinning up micro-services using Ruby/Kafka
38mEffective Data Synchronization between Rails Microservices
30mChange Data Streaming Patterns in Distributed Systems
42mData Streaming for Microservices using Debezium
33mMicroservices, Events, and Breaking the Data Monolith with Kafka
51mStreaming Platforms in Event-Driven Systems: Patterns, Pitfalls, and Performance