FlexGroup Volumes: A Distributed WAFL File System
Talkat USENIX ATC '19 · on Wednesday, 2019-07-10· at 14:40· USENIX ATC Track I: Grand Ballroom I–VI·
summary
FlexGroup turns many local WAFL volumes into a single distributed namespace by adding a remote links layer, a route cache with delegations for consistency, and ingest-time placement heuristics driven by a gossip of space/CPU. Lab benchmarks show metadata-heavy workloads pay a noticeable latency hit (approaching an always-remote case) while data-heavy workloads scale well. Field telemetry from over 1,000 customer FlexGroups across hundreds of petabytes finds over half have a coefficient of variation <1% and 78% under 5%, so automatic balancing largely succeeds.
AI-written summary · about
chapters (7)
- 0:00Introduction to flex groups
- 2:04History and requirements
- 4:45Design of remote access
- 6:54Operations and heuristics
- 10:42System evaluation
- 14:52Real-world performance
- 17:07Conclusion and Q&A
related talks
26mRunning FDB at scale at Wavefront
14mFalconFS: Distributed File System for Large-Scale Deep Learning Pipeline
42mProduction FS: Adapt or Die
21mNew Techniques to Improve Small I/O Workloads in Distributed File Systems
31mScaling Autonomous-Driving R&D: A 100 PB, 100 Billion-File Open-Source Elastic File...
41mUsing External Persistent Volumes to Reduce Recovery Times and Achieve High Availability on DC/OS
50mRunning Databases Efficiently at Scale - Case Study of Flipka...
05mProject Lightning Talk: Fluid Data Anyway, Data Anywhere, Data Anytime