Ceph: Lessons Learned from Emergency Support

on YouTube ↗44m

" summary

Emergency support cases across two–three years show concrete fixes and anti-patterns: a three‑OSD cluster is migrated to ~100 OSDs without downtime; RAID controller firmware reboots take 14 months to fully resolve; an OSD memory‑leak bug takes six months to land in release; PG autoscaler often misbehaves so teams disable and predefine PGs; avoid journal cache tiering (deprecated), prefer replication factor 3 with four nodes for safety, run burn‑in tests, monitor deep‑scrub CPU and monitoring DB growth, and use three monitors for ~500 OSDs.

AI-written summary · about

" chapters (25)

1. 0:00Introduction and background
2. 1:52Common Ceph setups and versions
3. 3:19Performance issues and migrations
4. 4:45MDS service and scaling challenges
5. 6:30Automation and data distribution
6. 8:17Hardware pitfalls and vendor issues
7. 10:02Multi-site and journaling trade-offs
8. 11:28Feature selection and workloads
9. 12:52PG Auto Scaler and performance
10. 13:55Cache tiering and deprecated features
11. 14:36Data loss risks and recovery
12. 16:01Lift and shift misconceptions
13. 17:26Monitoring and resource consumption
14. 18:30Large cluster strategies
15. 21:17Cluster sizing and support teams
16. 24:41Q&A: MDS count and active replay
17. 27:10Q&A: Minimum OSD and hardware sizing
18. 29:01Q&A: Replication factor and disk failures
19. 30:46Q&A: Detecting disk degradation
20. 32:18Q&A: Large HDD recovery challenges
21. 36:25Q&A: Scaling clusters manually
22. 37:51Q&A: Optimal monitor count
23. 40:27Q&A: Cache tiering risks
24. 42:37Q&A: HDD clusters with SSD caching
25. 43:19Closing remarks