Scaling the Namenode - Lessons learnt
summary
Common causes of NameNode failure: millions of small files bloating metadata, suboptimal heap sizing (rule‑of‑thumb: ~1 GB heap per 1M blocks), excessive logging and synchronous audit/getFileInfo calls, and co‑located heavy services. Actionable fixes include disabling getFileInfo audit logging and using async logging, enabling service RPC, congestion control and RPC call queues, and the data node lifeline protocol for clusters >200 datanodes. Also throttle fsimage transfer (example: 65 GB image froze nodes), reduce max replication (default 512 → ~10–20), and tune zkfc timeout (45s → ~120s carefully).
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:31Common Namenode problems
- 3:16Small files and heap issues
- 5:43Underutilized HDFS features
- 8:31Optimizing logging settings
- 11:17RPC performance improvements
- 15:11Host-level scalability tips
- 17:15CPU and kernel tuning
- 20:07Group lookup performance
- 22:58Failover and startup tuning
- 28:49Heap size calculation guide
- 31:37Managing block reports
- 33:06Controlling small files
- 35:35Resources and summary
related talks
44mBringing Down the Cost of Your OpenSearch Clusters
37mPopular Apache Cassandra Problems & How to Fix 'Em
45mMonitoring Your OpenSearch Cluster Effectively
20mFrom timeout to sub-second: solving scale dependent deadlocks in distributed systems
29mWhat breaks when Aerospike hits 6 million QPS
30mUnderstanding Bottlenecks & Conquering them
25mDeciphering Cassandra’s Prophecy: Preventing the Fall of Your Application
20mMongoDB size management