What it takes for stream data processing

on YouTube ↗53m

" summary

The talk explains what a full streaming data platform needs: durable event storage, reliable delivery of those events to downstream systems, and stateful stream processing that can maintain and manage processing state. It discusses technologies and design concerns around event stores, delivery mechanisms, and stateful processing components for building production-grade streaming systems.

AI-written summary · about

" chapters (13)

1. 0:00Introduction
2. 3:22Stream processing needs pyramid
3. 10:53Data ingestion types
4. 22:17EventHub choice
5. 27:43Flink and alternatives
6. 33:26Real-time ad campaigns
7. 41:39Data monitoring integrity
8. 43:04Flink docs issues
9. 44:52Redis role streaming
10. 46:38Ops vs analytics
11. 48:24Streaming databases
12. 50:11Flink CDC for small
13. 51:37Q&A

" related talks

• 45mData Stream Processing Concepts and ImplementationsMatthias Niehoff · Devoxx Belgium 2017
• 32mBig Data Processing : Where do we go next ?Roshan Naik · Flink Forward Global 2021
• 41mStreaming: an Alternative to CRUD & Batch ProcessingSamuel Roze · AFUP Day 2020 Lille
• 01h 14mPANEL - Streaming.Mary Grygleski, Dean Wampler, Zander Matheson, Oli Makhasoeva, Tim Berglund, Jerry Peng · Scale By the Bay 2023
• 41mReliability of Data Delivery in Stream ProcessingEvgeniy Ravnushkin · Saint HighLoad++ 2023
• 38mThe story of building a real-time streaming analytics systemRenat Idrisov · Backend Conf 2018
• 39mHigh-volume Data Streaming and StorageGleicon Moraes · QCon São Paulo 2019
• 31mBlueprints for common stream processing use cases with FlinkKostas Kloudas · Berlin Buzzwords 2018