Journey Through Real-Time Data Distribution with Kafka Pipelines
summary
A practical case study of using Apache Kafka for real-time data distribution: the talk explains ingesting database changelogs as continuous data sources, deploying Kafka Connect and Kafka Streams to stream changes to where they’re needed, and operating this pipeline on Kubernetes with monitoring. It covers problems encountered, lessons learned while building the data distribution platform, and suggested changes or takeaways for improving such pipelines.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:05Pre-change company problems
- 3:16Choosing Apache Kafka
- 6:52First proof-of-concept
- 9:41Connector challenges
- 13:41MongoDB and PostgreSQL integration
- 15:51Production transition and testing
- 17:57Cluster performance issues
- 21:38Schema approach change
- 22:42Platform adoption growth
- 27:39PostgreSQL scaling challenges
- 29:28User interface development
- 31:59Cluster expansion and alternatives
- 33:26New data extraction approach
- 34:53Summary and conclusions
- 37:25Q&A
related talks
46mKafka Streams at the Big Data Crossroads at Lightning Speed
19mAI Workflows with Kafka and Modern Architectures
30mDeploying SQL Stream Processing in Kubernetes with Ease
47mIn Real Time with Apache Kafka
50mApplication Design Patterns on Apache Kafka
31mEvent driven communication with Kafka on Kubernetes
35mBringing Your Data Pipeline into The Machine Learning Era
27mRunning Kafka in Kubernetes a practical guide