Operating Cloud-Native Kafka

on YouTube ↗25m

" summary

Overview of running Kafka at scale in the cloud with concrete tradeoffs and engineering steps: Confluent runs thousands of managed clusters on AWS/GCP/Azure, using multi-tenant on Kubernetes with a control plane for capacity and SLA management; networking requires VPNs, VPC peering, CIDR and load‑balancer limits per provider; tune throughput/latency/durability via acks=1 vs acks=all, linger, compression; use KSQL DB and over 100+ connectors; leverage cloud object storage for effectively infinite storage and regional placement for availability.

AI-written summary · about

" chapters (5)

1. 0:00Kafka and event streaming
2. 3:56Centralized event streaming
3. 8:50Scaling Kafka four ways
4. 15:50Cloud Kafka challenges
5. 24:03Parting thoughts

" related talks

• 36mTaming the cost of Kafka workloads in the cloudStefan Sprenger · Berlin Buzzwords 2024
• 48mCloud Native Event Streaming with Kafka & Open StandardsKasun Indrasiri · GOTO Chicago 2023
• 24mThe Cost of Apache KafkaJakub Scholz, Rajith Attapattu · Data on Kubernetes Day NA 2025
• 28mConnectivity Patterns for Cloud Native ApplicationsHugo Guerrero · Great International Developer Summit 2023
• 47mTackling Cloud Complexity - Recipes From Real World ProjectsMarek Urbanowicz · DevConf 2023
• 43mCloud Native - effective computing at scaleIgnacy Kowalczyk · GDG DevFest Sofia 2018
• 39mCloud Native Akka & Kubernetes The Holy Grail To ElasticityFabio Tiriticco · Scala Italy 2018
• 33mNATS: The Cloud-Native Challenger to Kafka's Messaging ThroneMohammed Moussaoui · BlaBlaConf 2025