Stream All the Things!
summary
Argues that data-centric and microservice architectures converge and compares four concrete tools: Spark, Flink, Akka Streams, Kafka Streams, with Kafka as the data backplane. Gives practical tradeoffs: Flink and Akka for low-latency per-event work, Spark for micro-batches, Kafka Streams for Kafka-centric transforms and windowed aggregations. Calls out a real constraint: payment auths need 10 milliseconds latency budget. Describes deployments, scaling, exactly-once semantics and where each tool succeeds.
AI-written summary · about
chapters (35)
- 0:00Introduction
- 1:20What is Hadoop
- 4:22Time vs Money
- 11:35The Paradox of Choice
- 12:45Criteria
- 13:12C
- 14:25Credit Cards
- 15:11Streaming Tools
- 15:54Latency Requirements
- 16:16Microbatch Processing
- 16:45Mixing
- 18:21Volume
- 19:28Partitioning
- 20:33Stream Processing
- 24:11Dataflow
- 25:23Scoring Data
- 25:53Pragmatic Considerations
- 27:33Google Dataflow
- 29:08Example
- 30:50Beam
- 31:45Akka streams
- 33:10Kafka streams
- 34:30Spark
- 35:11True Streaming
- 35:25Lambda Architecture
- 35:47Microservices and Fast Data
- 36:19Microservice Architecture
- 36:36Developer Experience
- 37:02Reactive
- 38:37Venn Diagram
- 38:59Scala
- 40:02Marketing slides
- 40:46Summary
- 41:03Jonas Poner
- 41:27Questions
related talks
36mStream All The Things!
33mKafka Microservices with Kafka plus Akka and Kafka Streams
37mBeyond Microservices: Streams, State & Scalability
44mKafka based Microservices with Akka Streams and Kafka Streams
30mEvent Streaming and the Future of Applications
55mThe data dichotomy: Rethinking data and services with streams
21mREST APIs vs. Data Streams: Which Is Right for Your Business?
59mBuilding Data Intensive Microservices with Apache Kafka