How to Pick Your Next Streaming Architecture
summary
Compares concrete systems and tradeoffs: Akka Streams + Scoobs is used at PayPal for a web crawler and yields about 10x speed versus a multithreaded Java version; Flink is natively true-streaming with low latency, iterative state and graph support while Spark uses micro‑batches for higher throughput; Kafka Streams exactly-once works under a closed‑world assumption; Apache Beam runners let the same pipeline switch engines; an IoT case reaches 1.5 million events/sec.
AI-written summary · about
chapters (11)
- 0:00Introduction and agenda
- 2:31Streaming system categories
- 4:17Apache Storm architecture
- 6:25Spark streaming approach
- 9:19Apache Flink design
- 11:06Apache Beam overview
- 13:38Kafka Streams benefits
- 14:43Akka Streams at PayPal
- 16:50Choosing criteria and scenarios
- 19:57Evaluation and decision process
- 21:04Q&A on monitoring and workflows
related talks
25mSPARK AND AKKA FOR BIG DATA SYSTEMS, IN PRACTICE
48mStreaming all the things with Akka Streams
32mBig Data Processing : Where do we go next ?
30mEvent Streaming and the Future of Applications
44mThe Art and Benefits of Computing Eventfully
47mOne Does Not Simply Query a Stream
33mKafka Microservices with Kafka plus Akka and Kafka Streams
41mStreaming: an Alternative to CRUD & Batch Processing