Kafka based Microservices with Akka Streams and Kafka Streams
summary
Compares two Scala libraries for Kafka-backed microservices with concrete code: a Kafka Streams pipeline that reads a raw topic and a model topic, trains models, serves scores and writes scored records (uses the new Scala API, KTable, and claims exactly once semantics via deduplication), versus Akka Streams using custom stages, backpressure and Alpakka connectors to talk beyond Kafka. Shows the model-training+serving topology, explains latency vs durability tradeoffs, and notes Kafka Streams is simpler for ETL while Akka gives finer-grained control.
AI-written summary · about
chapters (33)
- 0:00Introduction
- 0:40Streaming Context
- 0:50Reilly Report
- 1:17Architecture
- 1:38Data Backplane
- 2:00Kafka
- 2:47Partitioning
- 3:15Architecture benefits
- 4:00Kafka solves problems
- 5:30Kafka Streaming
- 5:50Spark and Flink
- 6:43Driver
- 7:10Pipelining
- 8:20Microservices
- 9:21Streaming alternatives
- 10:04Akka vs Kafka
- 11:00Events vs Messages
- 11:25Akka Streams
- 13:22Event Time vs Processing Time
- 15:17Exactly Once
- 16:19Case Dream
- 17:03Limitations
- 18:00Example
- 19:11Scala API
- 19:56Kafka Topics
- 21:31Kafka Stream Builder
- 25:15Running Microservices
- 28:22Akka Streams Example
- 30:36Kafka Streams Code
- 36:36Production Tradeoffs
- 37:41Latency
- 39:10Backpressure
- 40:02Wrapup
related talks
33mKafka Microservices with Kafka plus Akka and Kafka Streams
47mStream All the Things!
36mStream All The Things!
41mReducing Microservice Complexity with Kafka and Reactive Streams
59mBuilding Data Intensive Microservices with Apache Kafka
59mImplementing Microservices with Scala and Akka
46mTame Kafka streams with Clojure
46mComposable microservices for streaming analytics