Integrating microservices by accepting the data dichotomy
summary
Argues for accepting the data dichotomy by moving from shared mutable databases to a shared immutable log (Apache Kafka) plus local processing, using Kafka Streams state stores to do joins, aggregations and maintain local key/value views. Claims this yields service autonomy while keeping data accessible, with replayability, exactly‑once semantics and compacted topics to avoid infinite storage. Notes practical costs: petabyte replays need time or resources, retention/compaction policies, snapshots and standby replicas for fast recovery, and this pattern scales to millions of events per second in production.
AI-written summary · about
chapters (18)
- 0:00Intro
- 2:28What are microservices
- 5:03Scaling
- 11:31SSO
- 14:44The data dichotomy
- 15:04Dont share databases
- 16:13What can we do
- 18:24Data services
- 19:29Data erosion
- 20:18Data divergence
- 22:10Eventdriven architectures
- 22:27EDA
- 26:51Apache Kafka
- 35:09Data Manipulation
- 35:19Immutability
- 37:14Recap
- 41:09References
- 42:09Questions
related talks
55mThe data dichotomy: Rethinking data and services with streams
59mBuilding Data Intensive Microservices with Apache Kafka
55mThe hardest part of microservices: Your Data
37mBeyond Microservices: Streams, State & Scalability
51mData management in a Microservices world
42mCreating event-driven microservices: the why, how and what
47mThe Rise of Reactive Microservices
33mMicroservices, Events, and Breaking the Data Monolith with Kafka