On Track with Apache Kafka
summary
Public Network Rail train feeds ingest via ActiveMQ into Kafka Connect and lightweight tools like kafka‑cat, transformed and enriched with ksql and Avro schemas in the Schema Registry, then streamed to Elasticsearch, Neo4j, S3 and HTTP for alerts. The demo builds real‑time visualizations, per‑station configurable Telegram push notifications, joins movement and reference data, and persists Kafka as the source of truth. All code and configs live on GitHub and run end‑to‑end.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 3:40What is Kafka
- 5:07Getting data into Kafka
- 8:55Reference Data
- 10:02Kafka Collect
- 13:43Data Manipulation
- 15:09Stream Table Duality
- 18:02Schemas
- 19:58Schemers
- 21:02Schema Registry
- 24:15Transform Data
- 25:40Elastic Search
- 26:18Data Integration
- 29:54AWS Glue
- 32:40Events
- 33:54Alerts
- 37:16Resources
related talks
43mBuilding a community fountain around your data stream
50mA Journey in Space with Apache Kafka data streams from NASA
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
48mSwiss Transport in Real Time: Tribulations in the Big Data Stack
20mKeynote: Apache Kafka and the Rise of the Streaming Platform
36mKafka Streaming, the Pythonic Way
40mStreaming Geospatial Vector Data with Apache Projects
50mBuilding Data Streaming Platform with Apache Kafka