Uber HDFS Unit Storage Cost 10x Deduction
summary
Claims unit storage cost drops from >$500/TB in 2018 to under $50/TB by combining hardware and software changes: move to high‑density spindles (35× drives at 16 TB), per‑disk partitioning to mix hot/warm data, HDFS erasure coding (6 data + 3 parity stripes yielding ~1.5× overhead) and an access proxy that lets 2.x clients read EC data while offline converters rewrite replicated warm data. Built systems run in production and deliver the stated savings and similar IO utilization across old and new disk types. 10x unit cost reduction disk partitioning for mixed tiers erasure coding 6+3
AI-written summary · about
chapters (9)
- 0:00Introduction to HDFS cost reduction
- 2:21HDFS cluster architecture at Uber
- 3:4810x storage cost reduction journey
- 7:23Disk IO utilization and tiering
- 12:39HDFS erasure coding overview
- 17:22Adopting erasure coding at Uber
- 21:56Erasure coding access proxy
- 24:25Future plans and unified clusters
- 25:28Q&A session
related talks
53mCheaper, More Reliable, Simpler
12mFacebook's Tectonic Filesystem: Efficiency from Exascale
15mLESS is More for I/O-Efficient Repairs in Erasure-Coded Storage
19mConsolidating Storage Backends at Scale with Tectonic Filesystem
59mPractical Error Control Coding For Data Storage
17mOkapi: Decoupling Data Striping and Redundancy Grouping in Cluster File Systems
16mHadaFS: A File System Bridging the Local and Shared Burst Buffer for Exascale
30mScaling HDFS with Consistent Reads from Standby Replicas