Deep dive into Unbounded Data Processing Systems
summary
Compares concrete systems (Apache Samza, Spark Streaming, Apache Flink) and why Flink is chosen for Netflix’s cloud pipeline that processes over 700 billion events; explains event‑time, ingestion vs processing time, windowing, watermarks, checkpoints and exactly‑once semantics with state and savepoints; shows Kafka buffering and runtime design and claims measured wins: ~17% cost savings, ~40% better CPU and ~20% better network utilization versus Samza on their workload.
AI-written summary · about
chapters (8)
- 0:00Start
- 3:55Processing semantics
- 11:59Session Windows (Unaligned)
- 15:49Watermark example
- 19:40Functional Features
- 20:12Operational Features
- 31:02Spark Execution
- 40:37Keystone Management
related talks
36mStream All The Things!
41mFast Data - an Overview of the Concepts, Architecture and Technology of Streaming Analytics
44mThe Art and Benefits of Computing Eventfully
47mStream All the Things!
07mTransforming Big Data into Real-Time Product Insights
51mStreaming Platforms in Event-Driven Systems: Patterns, Pitfalls, and Performance
23mEvent-Driven Architecture: Unleashing responsiveness
42mRunning a Massively Parallel Self-Serve Distributed Data System at Scale