Heterogeneous pipeline processing with Kubernetes and Google Cloud Pub Sub
summary
Spacemaker replaces an expensive, slow AWS batch pipeline with a message-driven system using Google Pub/Sub, PostgreSQL state, and Kubernetes workers, claiming full team autonomy and heterogeneous resource support. A central broker implements a state machine with idempotent DB upserts to handle duplicates, scaling to a peak of ~2500 messages and adding nodes in 20–30 seconds via the cluster autoscaler. The pipeline works but needs careful transaction sizing and pod termination handling to avoid wasted work on scale-down.
AI-written summary · about
chapters (29)
- 0:00Intro
- 0:34Who uses Kubernetes
- 1:07Who is Hkon
- 8:04Pipeline steps
- 13:12Messagebased pipeline
- 13:52Cloud Pub Sub
- 15:12The occasional
- 15:47Lease period
- 16:52Multiple subscribers
- 17:42Why Pub Sub
- 18:57HTTP API
- 19:42HTTP Client
- 20:07Database
- 21:12State Machine
- 22:50Chaos Monkey
- 23:42Consistency
- 24:20Database Transaction
- 25:05Pipeline Broker
- 27:27Serial Execution
- 28:52Idempotency
- 31:24Summary
- 32:16Where we are
- 33:07Google Cloud Functions
- 34:47Lambda
- 35:20Functions as Service
- 36:15Kubernetes basics
- 37:19Live processes
- 38:54Pod manifest
- 45:51Recap
related talks
24mKubernetes Batch Processing at Scale - A Scheduling Perspective
01h 00mEvent-driven computing with Kubernetes
13mScaling Kafka on Kubernetes
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
36mKube-native Postgres [I]
28mBatch Processing with Spring Batch and Kubernetes
42mBuilding Scalable RAG Pipelines with Java and Kubernetes
26mBuilding Apache Spark Clusters on K8s for AI Workloads