Dynamic Big Data in the Cloud with Kubernetes
summary
Shows how to run Apache Spark, Flink and HDFS on Kubernetes using Terraform/Helm, operator patterns and node pools, and claims clusters can scale to bis zu 5000 nodes and cut runtime/costs by about 50% when tuned. Recommends the cluster autoscaler, spot/node-pool strategies and scale-to-zero but warns shutdowns and preemption break stateful jobs unless checkpoints and termination hooks exist. Describes NFS read-only mounts as a workaround for container storage, and concludes the approach works if autoscaling, labels and recovery are carefully engineered.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:12Cluster autoscaling basics
- 3:50Managed Kubernetes tools
- 7:09Operator pattern for apps
- 10:14Scaling strategies
- 13:08Spot instances benefits
- 16:32Migrating to new nodes
- 19:09Job scheduling tips
- 22:10Data processing workflows
- 23:45Summary
related talks
35mModern Big Data Pipelines over Kubernetes [I]
26mBuilding Apache Spark Clusters on K8s for AI Workloads
37mSpark and Flink Running Scalable in Kubernetes #bbuzz
31mKubernetes and the dynamic world in the cloud
29mDynamic Large Scale Spark on Kubernetes
41mBuilding & Operating a Large-scale HPC AI Cluster on Kubernetes
31mAn Adventure in Multi Cloud with Kubernetes and Distributed Data
28mProgress For Big Data In Kubernetes