Events... And what next?
summary
Event sourcing is implemented end-to-end: an HTTP Grape + Sidekiq collector accepts ~2.8M events/day (peaks 5.1M), persists to Elasticsearch event store (1.8B events, ~690GB), then live-processes updates into Postgres via Kinesis + Lambda while a full reprocessing job (RocksDB/Redis scratch, emit CSV, COPY into Postgres) rebuilds read tables. The pipeline survives spikes, accepts all events reliably, recovers from mistakes by reprocessing, and has run in production for ~18 months without total downtime.
AI-written summary · about
chapters (22)
- 0:00Introduction to events
- 1:16Starting with event sourcing
- 2:03Example ticketing system
- 5:37Event sourcing rules
- 7:23Updating relational database
- 7:44Event collector design
- 10:15Components overview
- 15:18Full reprocessing script
- 18:12Live processing with Kinesis
- 20:21Event store requirements
- 21:46Choosing Elasticsearch
- 24:06Elasticsearch scripting
- 26:14Kibana for event analysis
- 28:00GDPR compliance in event store
- 30:29Event store scale numbers
- 31:58Production deployment insights
- 32:18What not to event source
- 35:06Making mistakes safely
- 36:55Recovery and reliability
- 42:23Sources of inconsistencies
- 43:47Event store models
- 45:22Integration validations
related talks
54mEvent Sourcing
48mEvent Sourcing - what could possibly go wrong?
51mThings I wish I knew before I started with event sourcing
39mEvent Sourcing with Elixir
54mEventSourcingWithEvent Store
01h 07mAn Architecture with CQRS and Event Sourcing
43mEvent Sourcing dealing with an eventful past
01h 03mSurviving Long Event History