A SMART-er Ceph: Predicting Hard Drive Failure
summary
Uses the Backblaze SMART dataset and converts CSVs to Parquet, engineers six‑day window features (mean, std, coefficient of variation), and trains manufacturer‑specific models to predict three classes: good / warning / bad. A random forest model is easiest to integrate into Ceph and consistently outperforms ProphetStor on precision and recall for warning/bad cases. Pipeline handles class imbalance, uses Dask for preprocessing, provides Jupyter notebooks, a Kaggle‑style challenge, and a local Ceph module for testing and deployment.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 0:48Motivation
- 8:36Training Setup
- 16:25Data Analysis
- 17:55Fit
- 18:20Models
- 19:05Random Forest
- 21:35Results
- 23:30Demo
- 24:30Setup
- 25:10Start local cluster
- 26:00Testing
- 29:05Stop Ice Age
- 29:30The Goal
- 29:52Jupiter Notebooks
- 30:55Kegellike Setup
- 32:50Deep Learning Tools
- 33:45Overview
- 38:55Data Collection
- 42:25Power on Hours
- 45:00Domain Expert
- 45:40Model Feed
- 49:45Data Theater
related talks
31mFinding the Needle in the Haystack: Predicting Storage Device Failures in...
27mMaking Disk Failure Predictions SMARTer!
14mFailureMiner: A Joint Key Decision Mining Scheme for Practical SSD Failure Prediction...
31mAI, ML, and the Ceph Advantage: Scalable Storage for Smarter Workflows
24mUncovering Community and Project Insights Through Data Driven Methods
17mMulti-view Feature-based SSD Failure Prediction: What, When, and Why
32mPredicting Grid Faults Before They Happen: AI for Weather-Driven Grid Resilience
34mData Science beasts (failures) and where to find them