DIY: Data streaming platform with Kafka and Java

on YouTube ↗31m

" summary

Shows a runnable DIY streaming stack built with Docker Compose, Apache Kafka, Java clients and Kafka Connect, plus a JDBC connector and optional schema registry. Demo code generates a telescope feed into a telescope_feed topic, joins it with a celestial_metadata topic using Kafka Streams global table to split into planet/star/asteroid/alien topics, and exposes pitfalls: accidental typos creating missing topics, default auto-commit=true losing messages, and needing external configs for serializers and offsets. The pipeline runs in the demo but exposes fragile config and connector failures.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 1:08What is Kafka
3. 3:14Platform design principles
4. 8:13Kafka setup with Docker
5. 16:17Java producer implementation
6. 19:51Kafka streams processor
7. 27:16Data splitting and branching
8. 29:48Summary and use cases

" related talks

• 53mApache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!Viktor Gamov · Jfokus 2019
• 43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · LISA18
• 03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...F. Ramière, D Petisme · Devoxx France 2021
• 39mAnalysing Public Kafka Data from NASA SatellitesFrank Munz · Berlin Buzzwords 2025
• 39mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia Europe 2020
• 01h 00mIn Depth Kafka StreamsDaniel Hinojosa · Devnexus 2020
• 21mStreaming DataFrames: A New Way to Process Streaming DataTomas Neubauer · Conf42 Python 2024
• 41mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia November 2020