Learnings from Running Prometheus on Google Cloud Scale
summary
Google Cloud managed service runs Prometheus as a daemonset architecture using a custom operator and avoids ServiceMonitor CRDs; it operates millions of Prometheus instances ingesting trillions of samples per day. Memory-tuned flags, short retention and a lean agent mode without persistence reduce hundreds of terabytes of RAM at scale. Work includes PromBench remote-write benchmarks, VPA improvements that change limits without restarting, per-node memory limit ideas, remote secrets and plans to add OpenTelemetry/exporter integration.
AI-written summary · about
related talks
35mStateless Collectors For Stateful Data: Scaling Prometheus As a Node Agent
34mMonitoring Spinnaker with Prometheus Operator on GKE
24mHow to Be 10x SRE? A Deep Dive to Prometheus Operator
02h 43mPrometheus from A to Z
35mConfiguring Prometheus for High Performance
37mCloud-native Metric Monitoring with Prometheus
46mLeveraging Prometheus Service Monitor and Pod Monitor for Enhanced Kubernetes Monitoring
30mIntro: Prometheus