How we used event sourcing to organize data delivery and schema updates
summary
Describes using event sourcing to deliver structured data into a storage while allowing table schemas to evolve. Explains challenges of schema change management, describes an event-driven pipeline that records changes and applies schema migrations to keep data and structure consistent. Mentions concrete implementation details and operational considerations for maintaining updatable storage using events.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 3:16Existing solution problems
- 10:21Alternative approaches
- 17:24Schema format and data catalog
- 19:29Data delivery to storage
- 25:50Common errors and fixes
- 31:51Summary and solution applicability
- 32:53Q&A
- 44:02Consistency and load questions
- 45:54Team motivation and changes
- 48:19Breaking the pattern and final conclusions
related talks
25mFrom Guesswork to Governance: Bringing API Discipline to Data Streaming
23mEver Changing Data Model Schema Management For The Future
25mDealing with data in dynamic systems
49mTransforming Analytical Data with Event-Driven Architecture
39mHigh-volume Data Streaming and Storage
29mEvent streaming with modern data pipelines in a SaaS architecture
34mTales of Data Architecture Evolution
44mThe Changing Face of ETL: Event-Driven Architectures for Data Engineers