How we converted our Cortex data to TSDB blocks
summary
Describes converting hundreds of terabytes of Cortex chunk data into Prometheus TSDB blocks using three components: an index scanner, block builder, and scheduler that produces plan files per user-day from a full index scan. Block builders download chunks, deduplicate samples, sort series (moved to disk after hitting a 30 GB memory limit), write segment and index files, and emit meta.json; an early compaction-metadata bug was fixed. Result: all chunks become TSDB blocks and Bigtable instances are downscaled, conversion completed in days.
AI-written summary · about
chapters (12)
- 0:00Start
- 0:15Introduction
- 1:52Original Storage in Cortex
- 4:02Tsdb Blocks
- 4:51Segment Files
- 5:31Index in Tsdb Block
- 9:37Scanner Scans the Index Tables in Cortex
- 11:40Block Builder
- 15:31Meta Json Files
- 16:07Memory
- 16:35Horizontal Scaling
- 21:10Meta Json File
related talks
39mDeep Dive: Cortex
29mBeyond TSDB: Unlocking Prometheus with Parquet for Modern Sc...
20mIntro to Scaling Prometheus with Cortex
24mTSDB Data Compactions on Scale Without Headache: Now & Future
37mPrometheus Deep Dive
28mCortex 101: Horizontally Scalable Long Term Storage for Prometheus
30mTSDB: The Past, Present and the Future - Goutham Veeramachaneni, IIT Hyderabad
45mScaling Prometheus with Cortex, one time series at a time