On Data Locality in Kubernetes
summary
Experiments convert Tesseract OCR into microservice pipelines on a 3-node Kubernetes cluster (3 workers, 8 cores, 32 GB RAM, 50 GB disk) and compare remote object storage versus local caching and direct transfers. Local memory and disk caching achieve up to 4.4x and 2.1x higher read/write bandwidth versus remote, object stores are ~6x slower, data‑centric placement cuts end‑to‑end time ~2.6x (consolidated placement ~4.3x), and direct pod‑to‑pod data forwarding can speed stages up to 60x while Alluxio + CSI and prefetching supply namespace caching and persistence.
AI-written summary · about
chapters (10)
- 0:00Start
- 1:38Background
- 5:35Data intensive workflow: Tesseract
- 6:17Tesseract Microservice Benchmark
- 8:17Evaluation: Local PV vs. Remote PV
- 19:24Accelerating under storage data access
- 22:01Alluxio support architecture support
- 26:21Prefetch data with Distributed Load
- 28:33Alluxio CSI Driver v1.1.0
- 30:39Alluxio @ Microsoft
related talks
29mOn Data Locality in Kubernetes
05mLightning Talk: How to Bring Data Locality to I/O-Intensive Workloads on Kubernetes
45mEmbracing Big Data Workload in Cloud-Native Environment with Data Locality
35mEconomics of using Local Storage Attached to VMs on Cloud Providers [I]
31mThe Future of Data on Kubernetes: From Database Management To AI Foundation
07mData Gravity and Kubernetes: Managing Large-Scale Data Ingest With Minimal Latency
06m⚡ Lightning Talk: Colocating Data Workloads and Web Services on Kubernetes to Improve Resource Utilization
28mPersistent Storage with Kubernetes in Production - Which Solution and Why?