Scaling Databases and File APIs with Programmable Ceph Object Storage
summary
SkyhookDM extends Ceph object classes to run real database ops inside storage, implementing SELECT/PROJECT/AGGREGATE, indexing, and object-local row and column layouts using Google Flatbuffers and Apache Arrow. It adds HDF5 VFL-style backend support, can dynamically do row↔column transforms per object, and scales via generated names across 10,000 objects. Experiments on a 1.75B-row TPC‑H table show local transforms are far cheaper than pulling data out, distributed transforms cost more, and pushdown cuts client CPU and network significantly.
AI-written summary · about
chapters (20)
- 0:00Intro
- 2:47Skyhook Data Management
- 3:15This Talk
- 5:06Data management in Storage?
- 5:49Overview of Our Approach
- 7:05Pushdown Processing is an old concept
- 8:24What about data management?
- 9:56Skyhook DM Architecture
- 12:31Previously Row-oriented
- 14:07Support for Column Processing
- 14:47Processing Types
- 15:32How to Embed Semantics?
- 16:55Data Partition Metadata Required
- 17:49Scalable APIs
- 21:27ROOT access - obj access
- 22:23Python Interface for Scientists
- 23:09Physical Design Managment
- 23:38Physical Design Management Overview
- 26:18PROJECT before/after transform
- 27:28Scalability
related talks
02h 59mCeph Storage: a Caffeinated Primer
49mIntro to Ceph, the Distributed Storage System
34mModernizing Ceph Deployments at CERN: CephFS and Object Storage Across Data Centres
29mCeph Operations at Scale
21mNew Techniques to Improve Small I/O Workloads in Distributed File Systems
17mMAPX: Controlled Data Migration in the Expansion of Decentralized Object Based Storage
19mBuild Global-Scale Apps Using Globally Distributed Autonomous Databases
29mGlobal Deduplication for Ceph