Scaling Observability Stack for AIOPS
Talkat Observability Day Europe 2023 · on Tuesday, 2023-04-18· at 16:20· Hall 7, Room E | Ground Floor | Europe Complex·
summary
Intuit runs ~2,500 Kubernetes services and keeps Prometheus metrics for a week by adding Thanos components (query, store, query-frontend) with S3 long‑term storage and an internal ALB in front of Thanos query. Root causes are Prometheus/Thanos chunk pool and index pack sizing, solved by tuning chunk_size and index pack settings and by using Thanos query-frontend to split ranges. End-to-end load tests show ~30s average, 99th p 65s, stable CPU/memory on C5 reserved instances and total cost ≈ $3,100/month (ALB $204, S3 ~$2).
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:02Problem Statement
- 1:42Scaling Prometheus with Thanos
- 2:15Thanos Architecture
- 3:04Prometheus & Thanos
- 5:08Solution to Challenge - 1
- 6:15Solution to challenge - 2
- 7:29Challenges - 3
- 8:42Challenge - 4
- 10:45Final Solution
- 11:53Cost Analysis
related talks
26mPrometheus That Scales
31mHandling Billions of Metrics with Prometheus and Thanos
41mHigh Available + Scalable Prometheus with Thanos in Alibaba
36mRunning the Observability As a Service For Your Teams With Thanos
22mMonitoring the World: Scaling Thanos in Dynamic Prometheus Environments
27mObservability with Prometheus and beyond
25mMulti-Cluster Observability: What you need to survive with a fleet
39mAbsorbing Thanos Infinite Powers for Multi-Cluster Telemetry