Managing Distributed Workflows at Scale
summary
Shows a real Workflow Orchestration Engine built for Unbxd PIM that runs nodes as Docker images on Kubernetes Jobs and scheduled runs via Kubernetes CronJobs, uses Kafka listeners in long‑running deployments, and controls job lifecycles from code with the fabric8 Kubernetes client. Workflows pass compact metadata checkpointing between nodes so state resumes after failures, orchestrator queues triggers to avoid drops, TTL controller cleanup prevents pod bloat, and rate‑limiting via pod caps protects downstream databases — designed and proven to scale where the prior system did not.
AI-written summary · about
chapters (11)
- 0:00Introduction and overview
- 1:48Engine objectives and design
- 3:30Workflow logic and data flow
- 6:12Orchestration mechanics
- 8:07System architecture
- 9:29Kubernetes integration
- 11:50Implementation details
- 14:06Scaling and performance
- 15:36Rate limiting at scale
- 17:12Best practices and wrap-up
- 19:30Q&A session
related talks
29mHas Everyone Forgotten Application Workflows in Kubernetes?
27mCrafting Resilient and Scalable Systems with Orchestration and Choreography Patterns on Kubernetes
19mDistributed Workflows for Microservices-Style Applications [I]
17mRapidly Build Distributed Applications using Orchestration
36mOperator Framework to Manage Production Grade Stateful Workloads
28mOverview Of Challenges And Solutions For Orchestrating Applications To Multiple DC And Edge Clusters
36mSuccessful Kubernetes Development Workflows
34mScaling and Orchestrating “Good Bot” With Kubernetes