Stateless Collectors For Stateful Data: Scaling Prometheus As a Node Agent
Talkat KubeCon + CloudNativeCon North America 2022 · on Friday, 2022-10-28· at 14:55· Portside Ballroom (Room 260)·
summary
Argues for running Prometheus as a node agent (DaemonSet) that scrapes only co‑located pods and forwards metrics to a remote backend (Google’s Monarch), reducing single-server scaling pain. Implements new CRDs: podMonitoring / clusterPodMonitoring and rules + a RuleEvaluator deployment, plus an operator config singleton and system/public namespaces to enforce tenancy. Uses downward API, field selectors to cut API server watch load, and cites Prometheus benchmarks (4M series ≈ 4 cores / 25 GB) and Monarch serving over 2 trillion series. Works in production but secrets and centralized status remain open challenges.
AI-written summary · about
chapters (23)
- 0:00Start
- 0:24Outline
- 0:50Prerequisites
- 2:12Prometheus on Kubernetes
- 4:36Horizontally-scaling Prometheus
- 6:12Fix: Federation
- 7:02Fix: Remote Read
- 7:42Fix: Thanos
- 9:03Problem: rebalance shards
- 9:53Building a managed service
- 10:43Prometheus as a node agent
- 12:15Prometheus Service Discovery
- 12:48DaemonSet scaling considerations
- 14:33A new Prometheus operator
- 15:36PodMonitoring tenancy
- 17:09PodMonitoring CRD
- 18:12Rule evaluation
- 18:57Rules CRD
- 19:12Rules tenancy
- 20:25OperatorConfig CRD
- 22:32Minimize configuration surface
- 24:19Future direction
- 25:30Summary
related talks
06mLearnings from Running Prometheus on Google Cloud Scale
28mPrometheus Operators in the Wild
35mDoing Things Prometheus Can’t Do with Prometheus
02h 43mPrometheus from A to Z
37mMonitoring Kubernetes with Prometheus
24mCost-Efficient Multi-Cluster Monitoring with Prometheus, Grafana & Linkerd
32mDemocratizing Application Monitoring with Prometheus
36mDoing Things Prometheus Can’t Do with Prometheus