Bridging Big Data and Machine Learning Ecosystems: A Cloud Native Approac...
summary
Claims a working cloud‑native pipeline that removes Parquet serialization overhead by converting Iceberg/Parquet on S3 into in‑memory Arrow record batches served from a cache cluster (head + data nodes) over Arrow Flight, using Apache DataFusion for conversion and Kubeflow Trainer for orchestration. Dynamic sharding allocates precise offsets per worker so heterogeneous jobs (3 vs 5 workers) get only their slice, keeping GPUs fed at gigabytes/sec scale and reducing network IO; SDK and GitHub implementation are available.
AI-written summary · about
chapters (8)
- 0:00Introduction and overview
- 0:52ML training workflow
- 4:26Challenges in ML training
- 10:13The next-gen data stack
- 13:23Cache layer architecture
- 15:20Optimizing data transport
- 23:36SDK and implementation
- 25:22Q&A and closing
related talks
26mSpeed up Your ML Workloads With Kubernetes Powered In-memory Data...
26mUnlocking Scalable Distributed Training With Arrow Data Cache on Kubernetes
34mLeveraging Apache Arrow for ML Workflows
31mOpen Standards and Lakehouse AI Development
19mScalable Data and AI Solutions with Cloud-Native Architecture
31mMapping FHIR data to parquet for analytics and machine learning
39mTaming the language border (...)
20mCloud Native Patterns for Geospatial Analytics and AI