Back to fundamentals: from databases to event-driven platforms
summary
Describes converting a monolithic SQL database into a distributed commit log by using change-data-capture (CDC) to publish events to Kafka so multiple teams share a single source of truth. Gives a concrete implementation that streams CDC into Elasticsearch for search and materialized views, using Confluent Cloud and Schema Registry for governance, and discusses trade-offs like latency, retention costs, ordering and aggregation pitfalls, plus the need for schema and access rules.
AI-written summary · about
chapters (31)
- 0:00Introduction
- 4:03Complex info systems
- 6:11Databases' role
- 7:24Relational DB history
- 8:25DBs via log
- 9:52DB limits growth
- 10:35DB replication reads
- 11:43Event extraction CDC
- 13:07Traditional limits CDC
- 16:27Failure complex architectures
- 17:58Simpler solution log
- 20:281998 DB challenges
- 22:10Data export limits
- 22:31Replication messaging
- 24:14Physical DB limits
- 25:20Database partitioning
- 26:46Logical vs physical DBs
- 27:50Materialized views
- 28:58Distributed processing
- 30:03DBs vs streaming
- 31:51Streaming benefits
- 32:56Log as DB foundation
- 33:41Kafka as DB block
- 34:46Data governance
- 36:12Legacy Kafka CDC
- 37:17Distributed challenges
- 37:39CDC anti-patterns
- 38:43Kafka infinite retention
- 40:31Storage limits
- 40:53Business-tech alignment
- 41:40Conclusion
related talks
39mModern Event-driven Architecture: Adopting Data in Motion
27mNo More Silos: Integrating Databases and Apache Kafka
40mThe Changing Face Of ETL: Event-driven Architectures for Data Engineers
41mKafka as a Platform: The Ecosystem from the Ground Up
18mData Modernization Using Confluent and Yugabyte
30mChange Data Streaming Patterns in Distributed Systems
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
41mChange Data Streaming Patterns in Distributed Systems