Troubleshoot Compactor Backlog with Ease
summary
Compactors run as a cron job every five minutes and perform compaction, downsampling, retention in sequence so backlog usually appears in phase one when compaction can't keep up. Identify it by stagnant counters for iterations and downsamplings performed, degraded long-range query performance, or the v0.24 progress metrics that expose compaction progress. Fixes: scale vertically (more CPU), raise the compaction concurrency and downsampling concurrency flags, or scale horizontally by time ranges or by external-label sharding.
AI-written summary · about
chapters (6)
- 0:00Intro
- 0:16Thanos Compactor
- 1:58Identify the backlog
- 2:54Progress Metrics
- 3:24Scale Vertically
- 4:08Scale Horizontally
related talks
24mTSDB Data Compactions on Scale Without Headache: Now & Future
16mUnlock High-Frequency Deployments without Blowing Up Prometheus
24mScaling Argo CD: From Symptoms To Solution
29mThe osquery Agent. Performance Tuning @Scale
30mReliable Data Processing with Minimal Toil
28mResolving Outages Faster With Better Debugging Strategies
23mDesigning for High-cardinality Metrics
28mCommon Support Issues And How To Troubleshoot Them