Embracing Big Data Workload in Cloud-Native Environment with Data Locality
summary
Compares how mainstream Kubernetes container-attached storage exposes (or fails to expose) data locality, then presents Apache Hadoop Ozone’s network-topology features and an Ozone CSI plugin that provides locality-aware storage to the scheduler. Shows how using Ozone’s locality information enables Kubernetes to schedule big-data compute (demonstrated with Spark on Kubernetes) closer to data to improve performance, reliability and availability.
AI-written summary · about
chapters (16)
- 0:00Start
- 0:54Outline
- 2:01Cloud Native Environment
- 2:41Big Data Evolution in
- 4:27Cloud Environment - 1
- 10:24Cloud Environment - III
- 13:46Scheduling
- 15:41Rack aware access
- 17:21Short Circuit Read Namenode
- 18:01Apache HDFS in
- 20:09Locality in Big Data - Storage (Apache Ozone)
- 23:18Apache Ozone Overview
- 24:16Apache Ozone Storage Container
- 28:34Apache Ozone Topology
- 30:57Aware Container Replication
- 33:54Apache Ozone in Kubernetes Road Map
related talks
05mLightning Talk: How to Bring Data Locality to I/O-Intensive Workloads on Kubernetes
24mDynamic Big Data in the Cloud with Kubernetes
29mOn Data Locality in Kubernetes
25mCloud native data replication strategies
28mProgress For Big Data In Kubernetes
47mCloud Native and Sustainable, Reproducible Scientific Computing
07mData Gravity and Kubernetes: Managing Large-Scale Data Ingest With Minimal Latency
38mBeyond Kubernetes: Adapting To Spe...