Ingesting A Million Time Series Per Second On A Single Instance
summary
Claimed ingest rate exceeds 1.4M rows/sec on a single instance while serving queries concurrently. Implementation details: a columnar, time‑partitioned engine that writes timestamps sorted on disk so no secondary indexes are needed, Postgres wire compatibility plus a binary ILP ingest protocol, custom C++ JIT for filters, zero GC and memory‑mapped I/O, bespoke fast logger and I/O libraries. Real numbers: heavy users sustain ~100k/sec, a 1.6B‑row query runs ~557 ms, and CSV bulk ingest drops from 28 to 5 minutes using io_uring speedup 28→5 minutes.
AI-written summary · about
chapters (12)
- 0:00Introduction to QuestDB
- 1:11Identifying time series needs
- 2:56Indicators for time series
- 5:19Live demo and SQL features
- 11:14Performance design philosophy
- 14:21Ingestion and compatibility
- 15:33Engineering for speed
- 20:28Leveraging modern hardware
- 21:37Query optimization techniques
- 24:18Continuous performance gains
- 25:38Batch ingestion improvements
- 28:19Conclusion and resources
related talks
43mIngesting over 4 million rows a second on a single instance
39mTimescaleDB: Re-engineering PostgreSQL as a time-series database
51mPostgreSQL, The Time-Series Database You Want
51mBuilding a scalable time-series database on PostgreSQL
53mEfficient Time Series with PostgreSQL
49mPostgreSQL, The Time-Series Database You Actually Want
41mHigh-Performance Ingestion with Python and Swarm64DB
50mScaling Postgres With Some Help from Redis