Efficiently Updating Database State from Stream Processing Services
summary
Describes how Yandex Ads stores and updates complex entities (like advertising banners) in databases for streaming-processing services. Explains problems of keeping large, branched structures of many simple fields consistent and efficient, and discusses approaches to update those entity states from stream-processing services into the database.
AI-written summary · about
chapters (10)
- 0:00Introduction to stream processing
- 5:30Data compression optimization
- 7:15Columnar state separation
- 9:01Binary delta codecs
- 13:14State caching
- 17:31Cache transactional consistency
- 20:45Cold caching
- 21:26Aggregating columns in DB
- 25:00Optimization summary
- 26:44Q&A
related talks
39mHow Data Delivery Evolved for Yandex’s Ad Engine
40mStorage Architecture for Yandex.Direct Advertising Objects
23mStream Algorithms for Big Data Processing Tasks
41mStreaming: an Alternative to CRUD & Batch Processing
53mWhat it takes for stream data processing
21mPreparing Yandex search data — the library and processes we built
32mBig Data Processing : Where do we go next ?
48mThe internals of stateful stream processing in Spark Structured Streaming