Deploying SQL Stream Processing in Kubernetes with Ease

on YouTube ↗30m

" summary

Shows an end-to-end setup: Lenses for Apache Kafka plus the Stream Reactor connectors (almost 50) turn SQL-like configs into Kafka Streams apps, deployed via Docker/Helm to Kubernetes and monitored with Prometheus. Examples include a Cassandra connector used by Walmart and an Elasticsearch sink handling ~5 billion messages/day for British Gas. It explains Kafka Connect deployment patterns, Kafka rebalance problems and workarounds, RocksDB state backed to Kafka with StatefulSets for faster recovery, and a demo that spins processors to 10 pods processing ~500k messages each.

AI-written summary · about

" chapters (16)

1. 0:00Introduction
2. 3:12What is SQL Stream Processing
3. 3:46Kafka Connect
4. 4:21What is Kafka Connect
5. 5:06Kafka Connect Connectors
6. 5:38Kafka Connect Modes
7. 6:33Whats Good
8. 7:03Whats Not Good
9. 8:48Lets Connect
10. 10:58Execution Modes
11. 11:43Kafka Streams
12. 13:11Deploy
13. 13:34Site Optimizer
14. 15:46Lenses
15. 16:42Demo
16. 17:16Demo in Kubernetes

" related talks

• 35mStreams Must Flow: Fault-tolerant streams processing apps on KubernetesVictor Gamov · Montréal Cloud Native Day 2019
• 40mDeploying Kafka Streams Applications...Gwen Shapira, Matthias Sax · Scale By The Bay 2018
• 46mKafka Streams at the Big Data Crossroads at Lightning SpeedFrançois Sarradin, Jérémy Sebayhi · Devoxx France 2022
• 19mAI Workflows with Kafka and Modern ArchitecturesSreenivasaRao Jagarlamudi · Conf42 Prompt Engineering 2024
• 31mEvent driven communication with Kafka on KubernetesRaihan Khan · KCD Dhaka 2024
• 29mKafka on Kubernetes: Does it really have to be the hard way?Victor Gamov · Montréal Cloud Native Day 2019
• 27mRunning Kafka in Kubernetes a practical guideKatherine Stanley · Voxxed Days Microservices 2018
• 37mJourney Through Real-Time Data Distribution with Kafka PipelinesKonrad Szatan · JDD 2019