Reliability of Data Delivery in Stream Processing

on YouTube ↗41m

" summary

A presentation about ensuring reliable data delivery in streaming processing for Yandex.Market's offers storage, which stores billions of records and processes hundreds of MB/s. It describes the problems of ingesting partner data and propagating update chains across dozens of Market backends, and discusses the storage’s approaches and mechanisms to make these streaming updates reliable.

AI-written summary · about

" chapters (5)

1. 0:00Introduction
2. 4:52Stream processing architecture
3. 11:17Reliability challenges
4. 23:34Consumer transfer methods
5. 31:07Q&A

" related talks

• 53mWhat it takes for stream data processingMaksim Buylin · Saint HighLoad++ 2025
• 29mData Mesh Streaming Patterns: A Real-time PerspectiveAlok Pareek · Big Data LDN 2022
• 39mHow Data Delivery Evolved for Yandex’s Ad EngineDaniil Gavrilovskiy · HighLoad++ 2024
• 30mFive Challenges in Real-time Stream Processing and Five SolutionsFawaz Ghali · SREday 2023 London
• 39mHigh-volume Data Streaming and StorageGleicon Moraes · QCon São Paulo 2019
• 45mData Stream Processing Concepts and ImplementationsMatthias Niehoff · Devoxx Belgium 2017
• 32mBig Data Processing : Where do we go next ?Roshan Naik · Flink Forward Global 2021
• 21mPreparing Yandex search data — the library and processes we builtV.Agapitov, K.Khamitov · Saint HighLoad++ 2022