Pushing the Limits of Prometheus at Etsy
Talkat KubeCon + CloudNativeCon Europe 2025 · on Thursday, 2025-04-03· at 11:45· Level 1 | Hall Entrance N10 | Room G·
summary
A single Prometheus instance runs on 128 cores, 4 TB RAM and sustains peaks near 500 million series and 5 million samples/sec, but suffers remote_write lag and dropped scrapes; fixes include disabling compaction, writing continuous 1‑hour blocks, capping remote_write to 100 shards with larger batches, disabling HTTP/2 to avoid a 360 Mbps single‑flow limit, and constraining Go (GOMEMLIMIT ≈ 1 TB and tuned GOMAXPROCS), which reduces GC pauses and lets the server survive Black Friday traffic.
AI-written summary · about
chapters (9)
- 0:00Prometheus at Etsy intro
- 3:38Vertical scaling choice
- 8:16Remote write lag troubleshooting
- 13:38Remote write bottleneck
- 17:30Disabling compaction
- 23:46Garbage collection tuning
- 28:02Upstream improvements
- 29:08Lessons learned
- 34:10Q&A
related talks
33mIntro: Prometheus
32mPrometheus Deep Dive
36mDeep Dive: Prometheus
35mConfiguring Prometheus for High Performance
36mPrometheus - Intro, Deep Dive, And Open Q+A
39mHow To Take Prometheus Planet Scale: Massively Large Scale Metrics Installations
25mUnbreakable Prometheus: Leveraging Limits for Consistent Monitoring
37mJourney inside Prometheus: Architecture, updates, outlook