Embracing Big Data Workload in Cloud-Native Environment with Data Locality
summary
Argues that bringing compute to storage on Kubernetes measurably speeds big‑data jobs by using Ozone’s topology and a CSI plugin to expose locality to the scheduler. Shows a live stack: Ozone CSI plugin, Spark on K8s, and Ozone services (OM, SCM, datanodes with 5GB–16GB containers and replica placement across racks) and reports clusters at 250–1000+ nodes (theoretically 10,000) and object scale of over a trillion objects while demonstrating reduced network hops and faster reads.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:54Outline
- 9:20Big Data Evolution in ...
- 11:50Locality in Big Data
- 15:41Hadoop HDFS Locality
- 18:01Apache HDFS in Kubernetes
- 23:18Apache Ozone Overview
- 24:16Apache Ozone Storage Container
- 28:34Apache Ozone Topology
- 31:15Apache Ozone in Kubernetes
- 33:54Apache Ozone in Rubernetes Road Map
related talks
41mHDFS CSI Plugin: Speed Up Kubernetes in On-Premises Big Data Cluster
34mOn Data Locality in Kubernetes
35mModern Big Data Pipelines over Kubernetes [I]
29mOn Data Locality in Kubernetes
47mCloud Native and Sustainable, Reproducible Scientific Computing
24mDynamic Big Data in the Cloud with Kubernetes
05mLightning Talk: How to Bring Data Locality to I/O-Intensive Workloads on Kubernetes
06m⚡ Lightning Talk: Colocating Data Workloads and Web Services on Kubernetes to Improve Resource Utilization