Data @Scale Seattle - Kestutis Patiejunas
summary
Facebook builds a cold-storage stack deployed to a dedicated Oregon data center in 2014, using racks with 15 drives where one powered drive per 15 stays online to cut power and cooling and only ~6% of disks consume energy. Data is Reed-Solomon encoded (20/28 Reed-Solomon) and stored on raw sector writes (raw sector writes), enabling distributed rebuilds in under seven minutes and nine months from project start to production with four engineer-years invested; no user data loss since rollout and a planned Blu-ray robotic racks tier for longer-term archiving.
AI-written summary · about
chapters (9)
- 0:00Cold data storage at Facebook
- 1:08Evolution of storage hardware
- 5:59Designing for cold storage
- 7:29Software architecture principles
- 11:01System implementation and performance
- 13:58Data security and raw disks
- 15:43Future trends in storage media
- 18:26Blu-ray storage infrastructure
- 21:26Audience Q&A and conclusions
related talks
31mScaling Autonomous-Driving R&D: A 100 PB, 100 Billion-File Open-Source Elastic File...
36mHot Swap Your Datastore: A Practical Approach and Lessons Learned
32mLow Latency Serving of Offline Data: Efficient, Safe, and Reliable Data...
28mThe Ultimate DataHub for all your BigData needs
37mKEYNOTE: Data Architecture Turned Upside Down
19mDataEngBytes 2023 - SYD-T1-04 - Yash Sharma
45mBig Data and SSDs
PolarStore: High-Performance Data Compression for Large-Scale Cloud-Native Databases