InfluxDB storage engine internals
summary
Built a bespoke storage engine centered on a TSM tree that writes to a WAL, flushes memory into memory‑mapped TSM files with blocks of up to 1,000 points, and compacts files to keep series values contiguous. Timestamps support nanosecond precision; values use delta, double‑delta (Gorilla), bitpacking and snappy; tombstones and compactions handle deletes/updates. A separate on‑disk/in‑memory inverted index (TSI) uses Robin Hood hashing, HyperLogLog++ sketches, and is tested up to 500 million series on one box.
AI-written summary · about
chapters (8)
- 0:00InfluxDB storage engine
- 2:52Why new storage engine
- 7:08Requirements for engine
- 8:54TSM tree storage engine
- 19:39Indexing scheme and inverted index
- 23:52In-memory index challenges
- 34:54Robin Hood hashing in index
- 35:38Q&A
related talks
16mEnable better performance and reduce storage cost with InfluxDB 3.0
23mTime series database: Should I use one?
35mThe Internals of Metric Store
22mAt scale, everything is hard
01h 03mWiredTiger
28mTime series data science problems with InfluxDB
12mInfluxDB for Industrial IoT
13mNext-generation databases for next-generation workloads (PRT051)