Building a Distributed Task Scheduler With Akka, Kafka, and Cassandra

on YouTube ↗37m

" summary

PagerDuty builds a production Task Scheduler with Akka, Kafka, and Cassandra that ships in six weeks to production and executes millions of tasks per month. Kafka provides partitioned ordering by hashing an ordering ID, Akka runs lightweight per-ordering actors to serialize retries, and Cassandra stores durable state; the new system cuts Cassandra reads by 65% reduction in Cassandra reads, supports consumer rebalances for zero‑downtime scaling, and survives datacenter outage testing.

AI-written summary · about

" chapters (16)

1. 0:00Intro
2. 3:24The Problem - Use Cases
3. 5:14The Problem - Ordering
4. 6:10The Problem - Assumptions
5. 8:59Problems with The Old Solution
6. 10:21The New Hotness - Technology
7. 13:25Dynamic Load - Kafka
8. 17:35Dynamic Load - Cassandra
9. 19:54Datacenter Outages - Kafka
10. 22:02Datacenter Outage - Cassandra
11. 24:13Datacenter Outage - Service
12. 25:48Task Ordering - Kafka
13. 27:33Task Ordering - Service
14. 30:19Scheduler in Production
15. 31:59Did it solve our problems?
16. 34:13Questions?

" related talks

• 48mThe three paths of Scaling with AkkaFran Perez · Scala.IO 2017
• 41mAkka in Production: Our StoryEvan Chan · Pacific Northwest Scala 2013
• 55mExperience report on using Akka in production: pitfalls and rewardsFrançois Laroche · ScalaIO 2019
• 26mOperating Cassandra in Kubernetes with TemporalChristian Mauduit, Steven Seidman · Cassandra Summit 2023
• 33mKafka Microservices with Kafka plus Akka and Kafka StreamsBoris Lublinsky · Scala Days Berlin 2018
• 26mOptimizing Kafka Consumers with Kubernetes and KEDA...Shubham Badkur, Yuanzhe Liu, Simran Aggarwal · KubeDay Singapore 2023
• 31mWriting Reliable, Scalable, Fault Oblivious Code On K8s the Easy WayKendall Roden, Alice Gibbons · KubeCon + CloudNativeCon North America 2022
• 42mDistributed Data Stores on KubernetesAlena Hall · GOTO Chicago 2018