HDFS CSI Plugin: Speed Up Kubernetes in On-Premises Big Data Cluster
summary
Built a CSI-compatible HDFS plugin that mounts HDFS via an NFS gateway or via Ozone S3 path, deployed with per-node CSI node agents and a single controller, and used the TPC-DS benchmark to compare Spark on Kubernetes + HDFS against Spark on YARN + HDFS. The design preserves HDFS features (three replicas, snapshots but no dynamic snapshotting), prevents concurrent writers to the same file, and emphasizes operational tradeoffs for large clusters (thousands to tens of thousands of nodes) and controller placement. Results focus on architecture and limitations rather than raw speed numbers.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:58Outline
- 3:12Big Data History Cont.
- 3:33Big Data Stack
- 6:18Big Data Trend
- 8:45Benefit of Containerization
- 10:00Kubernetes Architecture
- 10:30Challenges
- 12:12CSI(Container Storage Interface)
- 15:19CSI Core Services
- 16:51CSI Advance Features
- 19:07Volume Lifecycle Volume Lifecycle
- 19:37Controller and Node Services
- 20:51Kubernetes Storages
- 22:17Kubernetes CSI Support
- 25:10PV, PVC and Storage Class
- 27:49Package and Deployment Suggestion
- 29:02Hadoop HDFS
- 30:04HDFS Cluster Scale
- 30:40Apache Ozone
- 31:44HDFS/Ozone as PV
- 34:25HDFS Characteristics as PV
- 36:08HDFS NFS Gateway CSI
- 36:40Ozone CSI
- 37:00Resources
related talks
41mHDFS CSI Plugin: Speed Up Kubernetes in On-Premises Big Data Cluster
48mCSI in Kubernetes
24mDynamic Big Data in the Cloud with Kubernetes
26mBuilding Apache Spark Clusters on K8s for AI Workloads
22mCSI Driver X FUSE Drivers: a Kubernetes Object Storage Solution for AI/ML Data Portability
46mStorage Requirements and Options for Running Spark on Kubernetes
39mSpark on Kubernetes: Best Practice and Performance
35mModern Big Data Pipelines over Kubernetes [I]