Designing a Query Queue for ScyllaDB
summary
ScyllaDB uses a custom query queue that handles variable-size items from hundreds of bytes to many megabytes and runs at about 10,000 ops/sec per vCPU (not millions). It replaces blind concurrency with a semaphore (e.g. 100 concurrent slots), per-query memory accounting and a two-level OOM killer that first stops new queries then kills all but one running query. It caches 1 MB pages (cache ≈ 2% node memory), tracks a subtask graph to admit queries only when workloads are I/O-bound, and exposes per-queue CPU/IO priorities.
AI-written summary · about
chapters (16)
- 0:00Introduction and background
- 2:35ScyllaDB architecture overview
- 7:14Why a query queue is needed
- 9:21Simple queue limitations
- 11:09Semaphore-based concurrency
- 12:13Memory accounting in the queue
- 13:17Paging queries and query cache
- 15:30Out-of-memory killer
- 17:19CPU vs disk concurrency
- 19:31Detecting CPU-bound queries
- 20:37CPU-bound workload handling
- 21:47I/O scheduling overview
- 22:08Query importance and multiple queues
- 23:36Field problems and fixes
- 24:39CPU concurrency override
- 25:02Conclusion and future work
related talks
41mPerformance & Fault Tolerance: Building a Modern Database
35mHow to Save on Scaling by Moving from Cassandra to Scylla DB
01h 02mBuilding state-of-art LSM-tree compaction in ScyllaDB
25mCache? Where we're going, we don't need cache!
44mScyllaDB where Cassandra meets Redis!
11mScyllaDB Tablets in Action
01h 02mBreak Out of Redis
43mIngesting over 4 million rows a second on a single instance