C.A.L.L.I.N.G. Now I'm Calling You, Calling You Now
Talkat KubeCon + CloudNativeCon Europe 2025 · on Friday, 2025-04-04· at 14:30· Level 0 | ICC Auditorium·
summary
eBPF daemonset collectors query the Kubernetes API for pod metadata and drive the API from tens/sec to ">2 million requests/s" with error spikes around 10,000 errors and doubled apiserver memory, taking a cluster down. Solution: replace per-node informers with a centralized gRPC cache that runs a few Go informer replicas, stores minimal fields, streams protobuf updates to agents, and cuts API load though startup refresh doubles cache CPU/memory briefly.
AI-written summary · about
chapters (6)
- 0:00Introduction and context
- 0:44Understanding eBPF basics
- 3:47The role of Kubernetes metadata
- 4:24Initial architecture and issues
- 9:09Analyzing the API performance
- 13:22Centralized cache solution
related talks
34mPanel: Improving and Managing Kubernetes at Scale
25mMore Churn No Problem: Lessons Learned Running Cilium in Production
36mMaking the Most Out of Kubernetes Audit Logs
42mPanel - Klustered: eBPF Edition
37mMonitoring Service Architecture and Health with BPF
29mKubernetes Failure Stories and How to Crash Your Clusters
25mScaling Data Platforms: Managing Growth from Zero to Billions of Transactions
37m10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You