Skyhook: Programmable Storage for Databases
summary
Skyhook pushes query work into Ceph OSDs using CLS plugins (C++/Lua) and per-node local RocksDB indices, claiming both I/O and CPU savings as storage scales. A TPCH lineitem test stores 1 billion rows (~140 GB) as 10,000 objects (~14 MB each), scales storage nodes from 1 to 16, and shows faster point and range queries when object-local predicates, projections, partial aggregates or regex run on the OSDs. Data is stored as sequenced flatbuffers per partition and uses Ceph primary-copy replication.
AI-written summary · about
chapters (10)
- 0:00Intro
- 2:09Ceph Distributed Object Storage
- 2:59Data Storage in Ceph Cluster
- 5:28Remote Processing Example
- 6:35Ceph Custom Object Classes (CLS)
- 7:53Skyhook - Data partitioning + format
- 9:30Skyhook - Remote Processing
- 13:05Skyhook - Ceph Extensions developed
- 18:44Skyhook - Key Takeaways
- 24:01Skyhook - Remote Indexing
related talks
43mCeph and Flash
31mRook: Intro and Deep Dive With Ceph
33mThe Challenge of Storing Small Objects on a Large Scale
12mSpanDB: A Fast, Cost-Effective LSM-tree Based KV Store on Hybrid Storage
29mCeph Object Storage Overview, Capabilities and Future Plans
39mBenchmarking Cloud Native Storage
07mRook: Enterprise Storage for Kubernetes
30mRook: Intro and Ceph Deep Dive