At scale, everything is hard
summary
Journey from InfluxDB 0.9 to a containerized 2.0 shows concrete choices: a custom on-disk engine called the Time-Structured Merge Tree, WAL via Kafka, object storage as the canonical store, and decoupled query compute with a new language flux plus a pushdown engine. Team scales from 12 to 90, codebase grows from ~35k to ~280k LOC, top customer drives ~50% of throughput, and Apache contributions (notably Apache Arrow) yield measurable zero-copy and AVX performance wins.
AI-written summary · about
chapters (12)
- 0:00Start
- 12:56Architecture
- 15:31Processing, Monitoring & Alerting
- 15:50Collection & Scraping
- 16:17Stateless Services
- 16:47Stateful Services
- 17:23Auto-Scaling
- 18:13Decouple Query from Storage
- 18:49Push Down Processing
- 19:14Optimize RPC
- 19:37Apache Arrow
- 20:22Sum 8,192 Values
related talks
16mEnable better performance and reduce storage cost with InfluxDB 3.0
12mInfluxDB for Industrial IoT
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
30mScaling Time-Series Data to Infinity: A Kubernetes-Powered Solution with Envoy
21mBuilding Distributed Platforms at Extreme Scale on Fast Changing Data
18mScaling Without Limits: KFIN's Journey to Distributed SQL
40mInfluxDB storage engine internals
22mIt’s all about the db - a journey to accommodate 100k events per second