Event-driven applications: Apache Kafka and Python
summary
Shows Python using kafka-python and Aiven Kafka to produce/consume JSON (and mention Avro/Protobuf + schema registry) with actual demos: creates topics, sets replication factor 3 and 2, creates partitions, sends three pizza orders and proves offsets, consumer groups and rebalancing work, and uses Kafka Connect (JDBC Sink) to auto-create a Postgres table that immediately receives records — connector runs and keeps populating the DB. Includes reusable notebooks and config files.
AI-written summary · about
chapters (16)
- 0:00Event-driven apps with Kafka & Python
- 3:25What events are and why they matter
- 6:13Apache Kafka explained
- 10:50Producing & consuming messages in Python
- 12:57Demo: Kafka & PostgreSQL setup
- 14:23Kafka producer in Python
- 15:28Kafka consumer in Python
- 17:36Log size & partitions
- 20:25Partitions and ordering
- 23:55Demo: Partitioning behavior
- 27:28Consumer groups & parallel processing
- 28:31Demo: Consumer groups
- 30:14Kafka with databases
- 31:58Kafka Connect pipelines
- 32:40Demo: Kafka Connect to PostgreSQL
- 36:31Summary & resources
related talks
26mEvent-driven applications: Apache Kafka and Python
45mEvent-Driven applications: Apache Kafka and Python
27mReal-time Event Streaming with Python
40mUsing Kafka in Your Python Applications
50mIntroduction to Apache Kafka
39mKafka as a Platform: The Ecosystem from the Ground Up
29mGo Big with Apache Kafka
01h 00mIn Depth Kafka Streams