Leveraging Apache Arrow for ML Workflows
summary
Claims concrete wins: Arrow enables 30x speedups (DuckDB + Flight vs Spark/Hive) by avoiding serialize/deserialize, Hugging Face caches several‑dozen‑GB corpora as IPC files and loads a 50–100 GB dataset into workflows using ~50 MB RAM via mmap, libraries expose zero-copy GPU buffers via DLpack/device data interface, pandas/polars/numpy interoperate without copies, and new canonical tensor types (Fixed‑Size and Variable‑Size) let end‑to‑end ML pipelines pass tensors and tabular data without costly conversions.
AI-written summary · about
chapters (11)
- 0:00Apache Arrow and ML workflows
- 1:30Why Arrow matters for ML
- 4:01What is Arrow
- 7:59Arrow in ML use cases
- 11:55Arrow Flight for efficiency
- 13:40Arrow for analytics
- 17:53Arrow in training and serving
- 21:27Arrow ML developments
- 24:59Integrating Arrow in ML
- 26:28Call to action
- 28:15Q&A
related talks
35mPractical Applications of Apache Arrow
33mArrow in Flight - New Developments in Data Connectivity
25mSpeed Up Data Access with PyArrow Apache Arrow Data is the new API
39mTaming the language border (...)
32mThe expanding Apache Arrow universe
33mFree Your Esoteric Data Using Apache Arrow and Python
36mEmbrace the chaos: Composable data systems with fewer asterisks
26mSpeed up Your ML Workloads With Kubernetes Powered In-memory Data...