Using SPIDER for sharding in production
summary
SPIDER is a MariaDB/MySQL storage engine that shards by using native table partitioning, supports cross-shard joins with join pushdown and aggregate pushdown, and uses two‑phase commit for multi‑node writes. Production deployments claim 200 billion rows per quarter across three Spider nodes and 14 data nodes, and Tencent runs 396 Spider nodes with 2,800 data nodes. VP (virtual partition) enables multi‑dimensional sharding; VP isn’t bundled yet. Roadmap items include force‑index pushdown, connection pooling, limit optimizations and automatic repair.
AI-written summary · about
chapters (11)
- 0:00SPIDER introduction
- 1:55SPIDER capabilities
- 4:29SPIDER in production
- 5:20SPIDER use cases
- 7:06SPIDER sharding
- 9:12Installing SPIDER
- 11:56Advanced SPIDER features
- 12:16Multi-dimensional sharding
- 15:52SPIDER roadmap
- 17:44SPIDER support
- 21:12Q&A
related talks
32mSharding for mission-critical, internet-scale apps
25mHow to shard MariaDB like a Pro
52mTo Shard or Not to Shard
27mPlug Into Analytics Connecting ColumnStore to Source Databases with Spider Engine
50mAerospike Technical Overview
06mProject Lightning Talk: Vitess in Production: Powering Applications at Massive Sca...
32mData @Scale Seattle- Eric Hwang introduced by Ganapathy Krishnamoorthy
25mThe future of databases: Combining operational and analytical data at massive scale