Kubernetes the Very Hard Way

on YouTube ↗41m

" summary

Datadog runs tens of Kubernetes clusters up to 2,000 nodes and describes control-plane design changes: separate etcd, API, scheduler and controller pools; use Vault for certificate rotation; and hit a production outage with 15,000 CSRs queued due to RBAC/token misconfig. Networking lessons: prefer native routing via BGP or cloud alias IPs over overlays, evaluate Calico/Cilium, use IPVS real server zeroing to avoid blackholing connections, and expect load‑balancer registration delays and scale limits on cloud LBs.

AI-written summary · about

" chapters (6)

1. 0:00Introduction to scaling
2. 3:57Designing the control plane
3. 7:22Securing the infrastructure
4. 18:58Efficient networking
5. 25:48Optimizing services and IPVS
6. 32:26Ingress challenges and closure

" related talks

• 44mKubernetes at Datadog ScaleAra Pulido · DevOps Barcelona Conference 2022
• 35mKubernetes, Lessons LearntKobi Biton · SREday London 2022
• 32mLaying the Foundation: Real World Kubernetes Deployment PatternsCraig Tracey, Josh Rosso · KubeCon + CloudNativeCon Europe 2019
• 34mCloudy With a Chance of Kubernetes: Going From One To Three Cloud ProvidersLaurent Bernaille, Maxime Visonneau · KubeCon + CloudNativeCon Europe 2025
• 37mKubernetesception: How to run a 1000 control planesRemy Rojas · ContainerDays Conference 2024
• 27mRunning Kubernetes in Production A Million Ways to Crash Your ClusterHenning Jacobs · Container Camp UK 2018
• 34mPanel: Improving and Managing Kubernetes at ScaleXiang Li, Corin Dwyer · KubeCon + CloudNativeCon North America 2019
• 35m10 Weird Ways to Blow Up Your KubernetesMelanie Cebula, Bruce Sherrod · KubeCon + CloudNativeCon North America 2019