Sharing responsively: Streaming analytics from sensitive data
summary
Builds a near‑real‑time anonymized analytics view by piping DB write‑ahead logs into a stream so analytics never query production and heavy queries don't impact users. Uses change data capture via Debezium into Kafka streams, with optional outbox pattern when schema coupling is unacceptable. CDC removes the dual‑write gap and keeps events consistent, but schema evolution, consent gating, hashing keys, and careful anonymization remain practical limits that the system handles rather than eliminates.
AI-written summary · about
chapters (41)
- 0:00Intro
- 1:00Data
- 1:40Moving to Amsterdam
- 2:09Simple applications
- 3:19Finding an answer
- 3:48Better solutions
- 4:01Generic solution
- 4:11Pros and cons
- 5:18Replication
- 5:50API
- 6:29Analytics API
- 7:33Conclusion
- 7:56Microservices
- 8:09Fragmented dashboard
- 9:21Data lakes ETL
- 10:33Adding an event bus
- 13:15Cons
- 15:50Change Data Capture
- 16:47How it works
- 18:05All their own spin
- 18:34Redhat
- 18:57Before and After
- 19:58Unlimited cons
- 20:40Data model coupling
- 21:20Outbox pattern
- 22:06Dual right
- 22:17Service team
- 22:28Three approaches
- 24:56Consent gate
- 26:07Anonymization
- 27:39Right to be Forgotten
- 29:05Event Based Data
- 29:31Analytics Stack
- 30:27Anonymizing Data
- 30:52Reserve Reversible
- 31:40Deanonymization
- 33:30Medical data
- 34:07Architecture
- 35:05Summary
- 38:25Thank you
- 38:37Questions
related talks
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics
39mApache Iceberg Merge-On-Read: Streaming CDC
11mBig Data Analytics: Immutability & Idempotence
20mChange Data Capture in Distributed Systems
01h 11mChange data capture pipelines with Debezium and Kafka Streams
45mModern Data Architecture
27mUnderstanding stream table duality using Kafka, Flink SQL and Debezium format
24mBreak Barriers In Real-time With Data Streaming