Running Apache Samza on Kubernetes

on YouTube ↗35m

" summary

Apache Samza runs natively on Kubernetes using K8s primitives for scheduling, storage and failure recovery, with a Kubernetes-backed scheduler replacing YARN or standalone deployment. The demo deploys Kafka, creates one job coordinator pod and two workers, reads Wikipedia edit streams, computes counts every 10 seconds and writes results to the wikipedia-states topic. State persists on an Azure File remote volume, Samza supports incremental checkpoints for several terabytes and thousands of cores.

AI-written summary · about

" chapters (17)

1. 0:00Introduction
2. 0:55Agenda
3. 1:37About Samza
4. 2:08Integrations
5. 2:58Typical Central Use Case
6. 3:47Features
7. 5:17Standard Terminology
8. 6:12Partition Tasks
9. 7:42Job Coordinator
10. 9:27Recap Communities
11. 10:05Workflow
12. 11:41Tasks
13. 13:46Wikipedia
14. 16:56Configuration file
15. 17:56Running a job
16. 22:59Other deployment options
17. 29:08Spark

" related talks

• 40mDeploying Kafka Streams Applications...Gwen Shapira, Matthias Sax · Scale By The Bay 2018
• 30mKafka on Kubernetes: From Evaluation to Production atShrinand Javadekar · LISA18
• 31mManaging Multi-Cloud Apache Spark on KubernetesIlan Filonenko, Aki Sukegawa · Kubernetes AI Day Europe 2022
• 26mProduction Grade Kafka on K8sAnand Iyer · Kubernetes Community Days Chennai 2022
• 33mKafka Operator: Managing and Operating Kafka Clusters in Kubernetes [A]Nenad Bogojevic · KubeCon + CloudNativeCon North America 2017
• 50mRunning Apache Kafka on Kubernetes and OpenShiftJakub Scholz · DevConf.CZ 2019
• 27mRunning Kafka in Kubernetes a practical guideKatherine Stanley · Voxxed Days Microservices 2018
• 26mBuilding Apache Spark Clusters on K8s for AI WorkloadsAnant Kumar · Conf42 Cloud Native 2025