Demystifying Stream Processing with Apache Kafka

on YouTube ↗55m

" summary

Argues that Kafka plus an embedded library changes stream processing tradeoffs: Kafka Streams is a lightweight, embeddable API (processor API + DSL) that uses Kafka topics and consumer groups for parallelism, local state backed by RocksDB plus a Kafka changelog for fault recovery, and supports event-time windowing and reprocessing via offsets and log compaction. Claims this reduces Lambda architecture complexity, fits existing deployment tooling, and notes at scale LinkedIn processes ~1.2 trillion messages per day using the same log-based platform.

AI-written summary · about

" chapters (19)

1. 0:00Intro
2. 7:36Myths about Stream Processing
3. 9:47Summary of Stream Processing
4. 11:51Stream Processing Challenges
5. 17:52Lambda Architecture
6. 22:32What is Kafka
7. 24:38Partitioning messages
8. 32:13Batch vs Stream processing
9. 33:46Stream processing as a consumer
10. 34:38Tapas Trains
11. 35:24Library vs Framework
12. 37:45Kafka Processor API
13. 40:11Kafka Application
14. 41:06Execution Model
15. 43:17Load Balancing
16. 44:17State Storage
17. 45:40Map to Physical Processes
18. 46:40Time
19. 50:53Kafka Connect

" related talks

• 45mApplications in the Emerging World of Stream ProcessingNeha Narkhede · GOTO Chicago 2016
• 50mApplication development and data in the emerging world ...Neha Narkhede · Berlin Buzzwords 2016
• 37mIntroducing Kafka Streams, new stream processing library ...Michael Noll · Berlin Buzzwords 2016
• 35mBuilding a Real-time Streaming Platform Using Kafka Streams...Jay Kreps · Data By the Bay 2016
• 33mKafka Microservices with Kafka plus Akka and Kafka StreamsBoris Lublinsky · Scala Days Berlin 2018
• 03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...F. Ramière, D Petisme · Devoxx France 2021
• 30mEvent Streaming and the Future of ApplicationsBen Stopford · JAX 2019
• 01h 29mEvent stream processing using Kafka StreamsFredrik Vraalsen · Berlin Buzzwords 2018