Kubernetes and the dynamic world in the cloud
summary
Argues that scaling short and big beats always-on clusters: running an 8x larger cluster for 4 hours yields similar cost and delivers results 8x faster. Shows GKE numbers: new node 30–45s, 3,000 pods (~1k nodes) in ~4 minutes, 18,000 pods in ~17 minutes. Recommends per-job clusters, cluster autoscaler, dedicated node pools, one pod per node, set min size to zero, use operators (notably the Google Spark operator) and Airflow integration for on-demand Spark/Flink deployments.
AI-written summary · about
chapters (27)
- 0:00Intro
- 0:53Businesses need dynamic scale
- 1:30operational challenges
- 2:31what you ideally want
- 3:19what scale up/down can achieve
- 6:46kubernetes in cloud
- 8:38k8s operators
- 9:33What are the benefits for big data
- 10:21same numbers on use with cluster-autoscaler
- 12:38cluster autoscaler scale down
- 14:06what does it mean for us
- 16:51what this means for us
- 17:37other k8s scheduler
- 19:04custom schedulers
- 19:49cluster auto scaler: add needed resources
- 22:18dedicated node-pools
- 24:08separate compute from storage
- 24:50a way to save network costs (across zones) / increase performance
- 25:50change compute via statefulset
- 27:04change compute via operator
- 27:16"cluster per job" on demand
- 28:07spark operators
- 28:37airflow and k8s
- 28:59flink operator
- 29:11storage hints
- 29:51image hints
- 30:16uninterruptible (GPU) jobs
related talks
24mDynamic Big Data in the Cloud with Kubernetes
29mDynamic Large Scale Spark on Kubernetes
37mSpark and Flink Running Scalable in Kubernetes #bbuzz
05mSponsored Keynote: The Cloud (And Your $$$) is Not Infinite: Dynamic Scheduling at Every Layer
29mRunning Kubernetes on 'small scale' clusters - lessons learned
25mExtending Kubernetes for AI | Lessons Learned From Platform Engineering
38mBeyond Kubernetes: Adapting To Spe...
07mKeynote: High Performance Computing on Google Kubernetes Engine