How we isolate streaming ingest from search using RocksDB
summary
Disaggregates streaming ingest and query by turning each JSON update into RocksDB key/value physical deltas on dedicated ingestion workers and streaming those deltas (including 64MB flush files and manifest updates) to query workers that only apply them into the RocksDB LSM mtable, cutting CPU cost by about an order of magnitude and keeping ingest latency subsecond. Uses S3 for durable writes and a shared hot SSD cache (shared hot SSD cache over S3) with prefetching so queries never hit S3; autoscaling can add capacity in ~30s, node recovery can take ~48min.
AI-written summary · about
chapters (7)
- 0:00Introduction and overview
- 3:56Cloud-native design principles
- 8:50Compute separation for ingest and query
- 17:24RocksDB replication for isolation
- 25:35Shared hot storage architecture
- 34:46Query compute separation benefits
- 35:11Q&A session
related talks
22mJoining Dozens of Data Streams in Distributed Stream Processing Systems
20mEnabling Machine Learning through Real-Time Data Processing
22mHow We Built SQL Rollups on Streaming Data
57mRocksDB
32mLow Latency Serving of Offline Data: Efficient, Safe, and Reliable Data...
29mAccelerating Performance: Mastering RocksDB for High Speed Data Processing
54mEVCache: Lowering Costs for a Low-Latency Cache with RocksDB
19mSILK: Preventing Latency Spikes in Log-Structured Merge Key-Value Stores