Stream Processing at the Edge with Apache Kafka

on YouTube ↗34m

" summary

Shows a working edge pipeline: an ESP32 sensor posting JSON over HTTP to an HTTP bridge, Kafka brokers run on a three-node Raspberry Pi k3s cluster with Strimzi, a Quarkus app uses the Kafka Streams API to compute one-minute windowed averages, MirrorMaker replicates aggregated topics to a central cluster, and Argo/Flux-style GitOps plus operators handle deployment and certs. Demo uses microPython on the device, k3s for Kubernetes, and real hardware, proving edge processing reduces latency and cloud egress while surviving intermittent connectivity.

AI-written summary · about

" chapters (14)

1. 0:00Stream processing intro
2. 2:32Stream processing in industries
3. 5:43Edge data processing
4. 7:30Edge environments for Kafka
5. 8:51Kafka for edge processing
6. 12:02Edge stream processing architecture
7. 15:50Kubernetes operators for edge
8. 18:02Demo setup overview
9. 21:57Edge Kubernetes cluster
10. 23:21Raw sensor data in Kafka
11. 24:04Kafka Streams API
12. 25:30Centralized edge visualization
13. 27:18Demo recap
14. 27:39Q&A

" related talks

• 39mAdvanced Stream Processing on the EdgeEduardo Silva · linux.conf.au 2020
• 03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...F. Ramière, D Petisme · Devoxx France 2021
• 25mAccelerating Data Processing and Consumption for Building Modern Apps at the EdgeMarilyn Basanta, Ed Byford · Kubernetes on Edge Day North America 2022
• 25mKubernetes IoT Edge Working Group: Using Event Driven Architecture at EdgeSteven Wong · KubeCon + CloudNativeCon North America 2020 Virtual
• 24mStreaming analytics on edge with Kuiper and KubeEdgeRocky Jin · Kubernetes on Edge Day Europe 2021
• 01h 29mEvent stream processing using Kafka StreamsFredrik Vraalsen · Berlin Buzzwords 2018
• 01h 00mIn Depth Kafka StreamsDaniel Hinojosa · Devnexus 2020
• 35mBuilding Smart Farms with Edge Computing Systems Using KubernetesSergio Mendez · Open Source Summit Latin America 2022