Troubleshoot Compactor Backlog with Ease
summary
Diagnoses why Thanos/Cortex compactor backlog forms (uneven shard load, slow object storage uploads, long compaction windows) and shows concrete signals to watch such as compactor_backlog metric and compaction duration per block exposed to Prometheus. Demonstrates using Prometheus queries, Thanos/Cortex debug endpoints and kubectl to measure backlog size and compaction concurrency, then applies fixes: increase compactor replicas, tune compaction schedule, and relieve storage bottlenecks, validating success by watching the backlog metric and compactions-in-progress collapse.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:15What are SLAs
- 0:47Error Budget
- 1:09Why
- 1:38User Interface
- 2:14Details Page
- 3:08Deployment Options
- 3:21Creating SLOs
- 4:01Grouping SLOs
- 4:12Pure
related talks
22mMonitoring the World: Scaling Thanos in Dynamic Prometheus Environments
26mPrometheus That Scales
27mWe Stopped Thanos from Snapping $100K from Infra
39mAbsorbing Thanos Infinite Powers for Multi-Cluster Telemetry
29mStreaming Recording Rules for Prometheus, Thanos, and Cortex Using...
27mHow Thanos Almost Snapped $100,000 from Our Infra Budget
15mScaling Observability Stack for AIOPS
33mStory of Correlation: Integrating Thanos Metrics with Observabi...