Stream All The Things!
summary
Compares concrete streaming engines and trade-offs: Kafka as durable event log, Spark and Flink for cluster-parallel processing, Akka and Kafka Streams as embedded libraries, and Beam as a semantics layer with runners. Gives latency budgets (tens of milliseconds for per-event, hundreds of milliseconds for interactive, seconds+ for micro-batches), highlights micro-batching vs per-event, discusses exactly-once semantics (deduplicated at-least-once), shows ETL, model training vs low-latency scoring, and argues streaming systems are converging with microservice architectures.
AI-written summary · about
chapters (4)
- 0:00Start
- 10:21Low latency? How low?
- 17:18Which kinds of data processing, analytics?
- 31:35A data app/microservice: • A single responsibility.
related talks
47mStream All the Things!
30mEvent Streaming and the Future of Applications
45mDeep dive into Unbounded Data Processing Systems
51mStreaming Platforms in Event-Driven Systems: Patterns, Pitfalls, and Performance
42mIt's like 10,000 streams when what you need is a queue
44mKafka based Microservices with Akka Streams and Kafka Streams
33mKafka Microservices with Kafka plus Akka and Kafka Streams
23mEvent-Driven Architecture: Unleashing responsiveness