Real-time Event Streaming with Python
summary
Demonstrates building real-time Python pipelines using Kafka (partitions, offsets, consumer groups) with confluent-kafka and mentions aio-kafka; shows simple patterns—filter, map, in-memory count/sum and a two-stream join implemented with dictionaries—and explains state/scale tradeoffs and changelog/RocksDB backing. Presents three small apps that produce a live tweet top tweeters leaderboard and a food-delivery ETA enricher that uses Quick Streams’ DataFrame integration. Gives sample producer/consumer code, explains when to use a library vs raw clients, and notes replay and scaling behaviors.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 0:47Event Streaming
- 2:08Broker
- 2:19Brokers
- 2:31Applications
- 2:47Libraries
- 3:20Other Libraries
- 3:36Kafka Logs
- 4:30Kafka Events
- 4:53Kafka Client Libraries
- 6:15Kafka Consumer Groups
- 7:08Kafka Clients
- 7:38Kafka Producer
- 8:27Kafka Consume
- 9:51Stream Processing
- 10:11Stateless Processes
- 12:21Stateful Processes
- 13:43Summing
- 14:17Join
- 16:32Code
- 17:08Challenges
- 18:19Libraries to help
- 18:48QuickStreams
- 21:05Questions
related talks
45mEvent-Driven applications: Apache Kafka and Python
38mEvent-driven applications: Apache Kafka and Python
26mEvent-driven applications: Apache Kafka and Python
20mStreaming DataFrames: A New Way to Process Streaming Data
26mBuilding A Python Service Using Open Source Components That Connect, Transform & Deliver Data In RT
36mKafka Streaming, the Pythonic Way
51mSimplifying Event-Driven Services with FastStream: A How-To Guide
40mUsing Kafka in Your Python Applications