rpdfs: Designing a High Performance File System
summary
rpdfs is a distributed file system built around per-NVMe device daemon processes, a lightweight rapidfs kernel client, and a quorum-based map daemon that enforces membership and RAID-like placement. It targets billions of files and up to exabyte-class archives, moves tens of GB/s to tape or S3, and provides coherence without a SAN by sending IO intents from clients to device demons. Core code is being relocated toward kernel.org and the kernel client is the current development focus; repos are not yet fully public.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:27Speaker background
- 3:57File systems overview
- 8:59Motivation for rpdfs
- 12:35Beyond shared disk storage
- 14:41Performance design goals
- 16:50rpdfs architecture components
- 20:44Development status and next steps
- 22:33Q&A
related talks
31mScaling Autonomous-Driving R&D: A 100 PB, 100 Billion-File Open-Source Elastic File...
19mFlatFS: Flatten Hierarchical File System Namespace on Non-volatile Memories
33mThe Evolution and Innovation of CubeFS
42mProduction FS: Adapt or Die
14mFalconFS: Distributed File System for Large-Scale Deep Learning Pipeline
29mCeph Object Storage Overview, Capabilities and Future Plans
17mPreview: Persistent Memory
16mOpenZFS