TSDB: The Past, Present and the Future - Goutham Veeramachaneni, IIT Hyderabad
summary
TSDB is the embedded storage engine behind Prometheus 2.0 and delivers concrete wins: 3–4x memory reduction, CPU for queried workloads drops from ~6 cores to ~1 core, and disk writes fall dramatically (Prometheus 1.0 wrote ~80 MB/s under churn). Implementation choices: write‑ahead log for instant restarts and planned replication, immutable blocks with hardlink snapshots that complete in milliseconds to seconds, tombstone-based granular deletes, fixes for histogram isolation, and Go optimizations (sync.Pool to avoid GC spikes, removal of unsafe “YOLO string” causing crashes).
AI-written summary · about
chapters (12)
- 0:00Start
- 5:46Insertion of Data
- 6:05Querying of Data
- 9:18Performance: On disk size
- 9:48Performance: Query Latency
- 12:30What's great: Isolation
- 14:29Gotchas: ALLOCS!
- 17:34Gotchas: yoloString!
- 19:26Gotchas: Files and Windows
- 21:54The future: WAL replication
- 23:51One more thing :
- 24:23Thanos
related talks
36mDeep Dive: Prometheus
06mLightning Talk
55mAdding ++ to Prometheus
23mPrometheus – What it is, What is New, and What is Coming
29mBeyond TSDB: Unlocking Prometheus with Parquet for Modern Sc...
01h 19mIntro + Deep Dive: Prometheus
30mLifecycle of a Sample in the Prometheus TSDB
26mWriting a TSDB from Scratch: Performance Optimization