Kubernetes Challenges at Lambda
summary
Lambda runs Kubernetes twice: a platform Kubernetes on owned hardware and a managed Kubernetes for customers, and both must handle integer-percent failure rates that produce multiple GPU failures per day. Clusters fill every switch port so there are no spares, repairs often require a full chassis power cycle making the PDU-as-cluster-resource, and long-running training jobs can keep nodes occupied for days. The team builds automated cordon/drain/reboot operators, a bare metal cluster API provider, continuous validation, and predictive-failure telemetry from thermals and power draw.
AI-written summary · about
related talks
06mKubernetes Love Machine Learning, Even on Private Cloud
29mKubernetes Workload Resiliency in Action: Beyond Basics
31mKubernetes For GPU Powered Machine Learning Workloads In Academia
27mRunning Kubernetes in Production A Million Ways to Crash Your Cluster
30mExplain How Kubernetes Works With GPU Like I’m 5
25mExtending Kubernetes for AI | Lessons Learned From Platform Engineering
22mBeyond Single-Cluster Limits: Scaling GPU Workloads Across Kubernetes With…
12mLightning Talk: Benchmarking Your Distributed ML Training on the K8s Platform