Chasing the Stream Processing Utopia
summary
LinkedIn ingests trillions of events per day into Apache Kafka and processes them with Apache Samza plus Brooklin change capture, using local RocksDB state and durable caches to make lookups local and achieve a 100X improvement over remote DB joins. The stack includes durable queues for shuffles, incremental checkpoints, a no-RAID Kafka setup, and open tools like Kafka Cruise Control and Apache Calcite/Beam integration for SQL and Python support, and it scales to PBs/day.
AI-written summary · about
chapters (26)
- 0:00Intro
- 2:34Stream Processing Technologies
- 5:49Types of Stream Processing
- 6:19Stateless Application: Simple Filtering
- 6:40Stateless Application: Data Enrichment
- 7:53Stateful Application: Aggregation
- 8:54Windowing
- 9:08Event Time-Based Processing
- 11:13Accuracy
- 11:57Exactly Once Processing - Ingestion
- 12:31Exactly Once Processing - Pipelined processing
- 14:07Scaling Ingestion
- 14:38Running Kafka @ Scale
- 16:43Kafka Cluster Management Woes Large deployment
- 17:08Kafka Cruise Control 1
- 17:42Cruise Control Architecture
- 17:52Scaling Processing: Challenges
- 18:20Typical Bottlenecks in Stream Processing
- 19:10Accessing Adjunct Data - Using Remote DB
- 20:42Accessing Adjunct Data - Using Local DB
- 22:44Maintaining Temporary State : Incremental Checkpoints
- 23:54Local State Gotchas!
- 25:12And Batch Sources..
- 25:49Stream Application in Batch
- 28:57Apache Beam
- 32:00Tools Ecosystem
related talks
29mApache Kafka and the Next 700 Stream Processing Systems
33mKafka streams at scale
41mBuilding an Open Source Streaming Analytics Stack with Kafka and Druid
41mStream Processor as a Database: Building Online Applications
29mData Acquisition: Overcoming Challenges
31mEvent Streaming with Kafka Streams
33mThe rise and fall of the full stack database
17mOvercome By Events - Apache Kafka And Event-oriented Architecture