Embracing Upstream Kubernetes in Web Scale Organization
summary
Runs single clusters of >5,000 nodes (targeting 10k) using virtual clusters with tenant control planes and a virtual node engine proxy so many tenants share one pool while keeping upstream APIs. Builds a custom scheduler, a static-operator for sidecars, and the open-source cruise workload controllers to enable in-place updates and hooks. Finds etcd fixes (HashMap change) speed up benchmarks to 1,000,000 KV ops in ~200s with <100ms latency and upstream API-server patches lift scaling to ~130 pod ops/sec on 100k pod tests.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:01Background
- 2:04High Level Architecture
- 3:57"Virtual Cluster" Hard Multi-tenancy Model
- 8:05Fix "Container Headache"
- 10:16Sidecar Operator
- 11:07Workloads Management
- 13:13Kruise: Kubernetes Workloads Advanced
- 16:10Scalability Matters
- 18:37Performance Benchmark Toolkit
- 20:14Discover Performance Bottlenecks
- 22:21Fix Performance issue etcd
- 24:28Fix Performance issue.kube-apiserver
- 29:44AKSS: A Tiger Team Dances with Upstream
- 31:46Wrap Up
related talks
50mScale Kubernetes to Manage 60K Nodes Through Architectural Extension
35mTo Infinite Scale and Beyond: Operating Kubernetes Past the Steady State
30mScaling Kubernetes to Thousands of Nodes Across Multiple Clusters, Calmly
34mPanel: Improving and Managing Kubernetes at Scale
03mFrom 5 to 1,300+ Clusters: Declarative Scaling for Private Kubernetes
33mKubernetes at Cruise: Two Years of Multitenancy
17mKeynote: Extending Kubernetes: Application Orchestration to 1000’s of Clusters in...
32mScaling Kubernetes Application to 1M Users