Consolidating Storage Backends at Scale with Tectonic Filesystem
summary
Consolidates Haystack, f4 and HDFS into a single datacenter-spanning filesystem called Tectonic that stores multiple exabytes per cluster and runs as a client-library driven microservice architecture. Uses chunk store with XFS files, linearizable key-value metadata layers, and RS-encoded blocks for throughput while converting small blob writes into triple-replicated partial blocks for low-latency uploads. Rollout spans about four years in two phases—software consolidation then cluster consolidation—and achieves Haystack-like latency and HDFS-like throughput while enabling multi-tenant IOPS sharing.
AI-written summary · about
chapters (8)
- 0:00Introduction to Tectonic
- 1:04Storage Challenges
- 3:12Legacy Architecture Issues
- 5:00Tectonic Design Goals
- 7:12Architecture and Scaling
- 10:16Performance Optimization
- 12:58The Rollout Journey
- 15:50Results and Conclusion
related talks
12mFacebook's Tectonic Filesystem: Efficiency from Exascale
42mApache Ozone - State of the Union
21mBuilding Distributed Platforms at Extreme Scale on Fast Changing Data
32mElevating Scalable Object Storage: A Comprehensive Exploration of Ozone’s Trailblazing Capabilities
21mNew Techniques to Improve Small I/O Workloads in Distributed File Systems
34mModernizing Ceph Deployments at CERN: CephFS and Object Storage Across Data Centres
16mHadaFS: A File System Bridging the Local and Shared Burst Buffer for Exascale
22mEveryone Loves File: File Storage Service (FSS) in Oracle Cloud Infrastructure