Object Storage Driven Machine Language Workloads
summary
Argues that designing pipelines around object storage removes painful data transfers and enables scalable, auditable ML workflows; demonstrates deploying MinIO and TensorFlow on Kubernetes, shows using the S3 API, Kubeflow/Argo-ish glue, and notebook-to-pipeline tooling to store intermediates and hyperparameters. Cites a 4.71 TB LLaMA dataset (≈800B documents) as an example and notes distributed training scale like 2000 GPUs for a week to justify treating storage as the cheap, central tier.
AI-written summary · about
chapters (11)
- 0:00ML pipelines and object storage
- 1:47Large-scale ML training challenges
- 3:53Compute vs storage separation
- 5:58Modular ML pipelines with MLOps
- 7:21Local dev with notebooks and Elida
- 10:50LLM training with object storage
- 12:33Demo: pipeline design with Elida
- 14:39Pipelines to distributed training
- 16:44Tracking ML experiments
- 19:34MLOps and model lifecycle
- 21:42Performance and Q&A
related talks
35mWriting Machine Learning Pipelines Against Object Storage- 35mWriting Machine Learning Pipelines Against Object Storage
31mBuilding a Data Infrastructure for AI/ML
37mHow to Accelerate Model Training and Eliminate the I/O bottleneck for the Cloud
26mMachine Learning Meets K8s
29mTeam Teardown: MLOps in the Cloud for IoT
41mInfrastructure for ML Applications
33mBuilding Resilient (ML) Pipelines for MLOps