Scaling Kafka in the Cloud

on YouTube ↗11m

" summary

Operational details and outcomes matter: Aiven runs 500 managed Kafka clusters across six clouds and 80 regions, sized from 3 to 30 brokers, and automates lifecycle operations so clusters undergo continuous roll-forward updates and live scaling without manual intervention. A management plane plus per-broker agents perform provisioning, automated partition placement and migrations, use ephemeral disks and immutable infrastructure, and claim high uptimes measured by reduced incident windows and successful large-scale migrations and version upgrades.

AI-written summary · about

" chapters (6)

1. 0:00Introduction
2. 0:38Who is Heikki
3. 1:32About his company
4. 2:32About Kafka
5. 3:09Benefits of Kafka
6. 5:24Architecture

" related talks

• 49mFour Different Ways of Working with Kafka on AzureHans-Peter Grahsl · Global Azure Austria 2021
• 35mKafka Deployment at ScaleOleksandr Nitavskyi · Highload fwdays'19
• 13mScaling Kafka on KubernetesShalini Katyayani Koney · Conf42 Kube Native 2025
• 27mRunning Kafka in Kubernetes a practical guideKatherine Stanley · Voxxed Days Microservices 2018
• 26mProduction Grade Kafka on K8sAnand Iyer · Kubernetes Community Days Chennai 2022
• 29mKafka on Kubernetes: Does it really have to be the hard way?Victor Gamov · Montréal Cloud Native Day 2019
• 24mMake Your Kafka Cluster Production-ReadyJakub Scholz · Data on Kubernetes Day North America 2023
• 42mProcessing Millions of Events per Second with Apache Kafka on KubernetesRasmus Selsmark · DevOps Pro Europe 2026