Unlocking Scalable Distributed Training With Arrow Data Cache on Kubernetes
summary
An Arrow-based cache sits between object storage and training, decoupling preprocessing and serving columnar batches with zero-copy Arrow Flight so PyTorch/TensorFlow/JAX jobs read in-memory Arrow records instead of serializing Parquet each epoch. Implementation uses Apache Iceberg + Parquet as the source, a Rust runtime, Kubernetes primitives (JobSet, leader-worker set) and can run with Kubeflow/Volcano; a demo populates shards (worker0 index0, worker1 index5000), fine-tunes distilled GPT2 on IMDb and uploads checkpoints to S3. No GPU-direct memory yet and end-to-end speedups are expected but not yet quantified.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:06MLOps and Kubernetes challenges
- 3:10Arrow Data Cache solution
- 6:42Cache architecture and Arrow internals
- 9:33Data storage and zero-copy transport
- 12:22Kubernetes integration and framework support
- 13:05Live demo setup
- 16:03Data retrieval and training execution
- 19:21Future roadmap
- 20:27Q&A
related talks
26mSpeed up Your ML Workloads With Kubernetes Powered In-memory Data...
33mBridging Big Data and Machine Learning Ecosystems: A Cloud Native Approac...
34mLeveraging Apache Arrow for ML Workflows
01h 02mApache Arrow: High-Performance Columnar Data Framework
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed Caching
06mInterview
45mAccelerating OpenSearch With Streaming: Apache Arrow, Flight, Data...
39mTaming the language border (...)