Kafka Streaming, the Pythonic Way
summary
Concrete claim: Kafka is a source of micro‑batches that can be deserialized into Arrow-backed DataFrames so the same DataFrame code runs against Parquet or live topics. Implementation uses a Rust-backed driver (Kafka RS) or confluent-kafka plus Apache Arrow for zero-copy exchange, Polars for JSON/Protobuf decoding, and Beavers to orchestrate microbatches, replay aligned history (Kafka default retention 7 days) and stateful windows. The stack is built and demonstrated to catch up quickly on restart and achieve production throughput.
AI-written summary · about
chapters (14)
- 0:00Kafka streaming in Python
- 3:10Kafka as real-time DB
- 8:31Live demo: Coinbase streams
- 12:29Kafka in Python challenges
- 13:36Reading Kafka data in Python
- 16:25Historical data replay
- 19:56Rust for Kafka drivers
- 21:20Kafka-RS introduction
- 23:24Stateful computation
- 25:53Beavers stream orchestration
- 27:18Live demo: GBP price filter
- 28:42Windowed aggregations
- 29:43Kafka in Python production
- 34:37Conclusion and key takeaways
related talks
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
27mReal-time Event Streaming with Python
20mKeynote: Apache Kafka and the Rise of the Streaming Platform
45mEvent-Driven applications: Apache Kafka and Python
29mUnified batch and stream processing in python
38mEvent-driven applications: Apache Kafka and Python
50mA Journey in Space with Apache Kafka data streams from NASA
50mBuilding Data Streaming Platform with Apache Kafka