Data Mesh Streaming Patterns: A Real-time Perspective

on YouTube ↗29m

" summary

Describes three concrete streaming patterns—real-time data integration, data-as-products-in-the-pipeline, and reverse ETL—and the practical pieces required to make them work: CDC from databases, handling before/after images and missing keys, checkpointing for state and failure recovery, and delivery guarantees like exactly-once. Mentions implementations using an in-memory layer (ZeroMQ) and a persistent backbone (Kafka), CDC-to-warehouse connectors (Snowflake/BigQuery), and a retailer storefront achieving sub-200ms end-to-end.

AI-written summary · about

" chapters (6)

1. 0:00Introduction to data mesh
2. 2:46Data mesh core principles
3. 10:08Shifting to real-time
4. 13:30Streaming patterns
5. 17:54Implementation complexities
6. 25:47Stream architecture and summary

" related talks

• 47mExpert Panel: Current & Future State of the Art in Real-Time Data InfrastructureFlink Forward Global 2021
• 40mA Data Mesh Of StreamsBen Stopford · State of Data Mesh 2022
• 45mModern Data ArchitectureAlexey Grishchenko · JavaDay Kyiv 2015
• 41mChange Data Streaming Patterns in Distributed SystemsGunnar Morling, Hans-Peter Grahsl · Flink Forward Global 2021
• 30mChange Data Streaming Patterns in Distributed SystemsGunnar Morling, Hans-Peter Grahsl · Berlin Buzzwords 2021
• 45mData Stream Processing Concepts and ImplementationsMatthias Niehoff · Devoxx Belgium 2017
• 36mChange Data Streaming Patterns for Distributed SystemsGunnar Morling · code.talks 2022
• 42mPrinciples and Patterns for Streaming Data AnalysisGalder Zamarreño · Voxxed Days Ticino 2018