TSDB Data Compactions on Scale Without Headache: Now & Future
summary
Explains how TSDB compaction merges index files and trunks (time-series chunks), handling horizontal (time) and vertical (overlap/backfill) merges, tombstone deletions, and planning for object-store workflows that fetch metadata, plan, download, compact, upload, delete. Describes a modifier interface added to rewrite blocks for relabeling, downsampling (example: 15s → 1m), and retention changes, an index size limit planner and replica_label deduplication using a penalty-duration algorithm; blocks flush from head to disk every two hours.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:37Definition
- 1:03TSDB Block
- 1:57meta.json
- 2:28Index & Chunks
- 4:10Two Blocks
- 6:35Compacted Block
- 6:51Vertical Compaction
- 8:03Query Blocks
- 9:09Tombstones
- 10:34Planning
- 11:44Thanos Compactor
- 14:36Prometheus HA Deployment
- 14:59Duplicate Data
- 15:54Deduplication Label
- 17:53Custom Vertical Chunks Merger
- 18:16Block Rewrite
- 18:59Relabel Block
- 20:32Extending TSDB Compaction?
- 20:47Compaction with modifiers
- 21:16Merge Blocks With Modifiers
- 21:39Brainstorming!
related talks
30mTSDB: The Past, Present and the Future - Goutham Veeramachaneni, IIT Hyderabad
23mHow we converted our Cortex data to TSDB blocks
26mWriting a TSDB from Scratch: Performance Optimization
15mAn Efficient Cloud Storage Model with Compacted Metadata Management for Performance...
01h 02mBuilding state-of-art LSM-tree compaction in ScyllaDB
51mSix New Important RocksDB Features And Planned Works
49mHidden Gems in SQL Server 2022 Database Engine
34mWhat We Talk About When We Talk About Disk IO