Running Large-Scale Stateful Workloads On Kubernetes at Lyft
Talkat KubeCon + CloudNativeCon North America 2019 · on Wednesday, 2019-11-20· at 10:55· Ballroom Sec 20AB - San Diego Convention Center, Upper Level·
summary
Lyft runs stateful Spark/Flink and ML pipelines on a homegrown platform called Flight and drives tens of thousands of concurrent jobs and tens of millions of containers per month. Flight uses a Kubernetes CRD-based control loop with a Flight propeller operator that hit API‑server/etcd and pod GC limits at scale. Fixes include resource quotas, heterogeneous node pools, a version cache plus CRD subresource updates and offloading large specs to blob storage, cluster autoscaler tuning, queue‑aware scheduling, spot-instance pools, and a multi-cluster failover setup that proves resilient.
AI-written summary · about
chapters (17)
- 0:00Start
- 1:19What are typical stateful workloads
- 2:08Stateful Workloads Lyft
- 3:15Introducing Flyte
- 4:28Concepts in Flyte
- 5:36Architecture Overview
- 6:28Dataplane Zoom in
- 8:04Status of a Workflow
- 8:39Ecosystem
- 9:11Bursty Load
- 10:08Need for Speed and Flexibility
- 11:09Observability & ROI
- 12:35Scale: K8s Cluster
- 16:20Scale: State Management
- 18:39Multi-Tenancy
- 22:08Performance
- 23:36Cost Optimization
related talks
17mKeynote: Extending Kubernetes: Application Orchestration to 1000’s of Clusters in...
32mMigrating 2 million CPU cores to Kubernetes
25mExtending Kubernetes for AI | Lessons Learned From Platform Engineering
06mKubernetes Challenges at Lambda
48mCloud FinOps & Kubernetes Optimisation at Scale
06mLightning Talk: OpenKruise: Advancing Kubernetes Workload Auto...
29mKubernetes Workload Resiliency in Action: Beyond Basics
12mKeynote: Building Enterprise AI/ML Platform on Kubernetes