Taking Care of Your Control Plane With API Priority and Fairness and Resource Quotas
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 11:15· Level 1 | Hall Entrance S10 | Room C·
summary
Datadog runs APF and quota tuning across hundreds of clusters (thousands of workloads, hundreds of thousands of pods on tens of thousands of nodes) and shows concrete configs: add flow schemas per namespace and for humans, create constrained priority levels, set borrowPercent=0 and lendablePercent=100 to stop unwanted seat borrowing, and set maxRequestsInFlight per core. They build a dynamic per-namespace controller for Resource Quotas using partial metadata (4x less memory) and a 409-retry workaround for admission conflicts; these changes stop control-plane overloads.
AI-written summary · about
chapters (10)
- 0:00Introduction and context
- 2:14Real-world incident examples
- 4:56API priority and fairness
- 9:07Tuning priority levels
- 12:53Advanced flow control tuning
- 16:06Autoscaling considerations
- 18:17Introduction to resource quotas
- 20:53Building a custom quota manager
- 24:12Controller development lessons
- 27:33Fixing upstream conflicts
related talks
38mInside Kubernetes Resource Management (QoS) – Mechanics and Lessons from the Field
20mBetter Resilience Through Self Tuning
10mNever Go Limitless – On the Importance of ResourceQuotas at the Edge
34mScaling Kubernetes up to 1,000 Control Planes - Dario Tranchitella
22mSpreading Apps, Controlling Traffic & Managing Costs in Kubernetes
25mAvoiding Overload in Distributed Systems
34mSpreading Apps, Controlling Traffic & Optimising Costs in Kubernetes
34mUnlocking Performance: Topology-Aware CPU Scheduling W...