Building GPU-Accelerated Workflows with TensorFlow and Kubernetes [I]
summary
A runnable Kubernetes pipeline uses containers, an object store and Pachyderm data versioning to wire pre‑processing, TensorFlow training on GPU and post‑processing together; pipelines are defined declaratively and trigger automatically when commits arrive. A satellite→map style‑transfer model trains in 11 minutes on CPU but about 1 minute on a p2.xlarge GPU after adding a GPU resource limit; provenance records which data, container image and node produced each result so the workflow is reproducible and schedulable.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:43Agenda
- 2:52Managing the Pipeline
- 3:54Preprocessing
- 4:55Model Training
- 6:43GPU Integration
- 7:50General Picture
- 8:40Kubernetes
- 13:00Extra Sugar
- 14:54Pachyderm
- 15:46Data Versioning
- 19:43Demo
- 23:16Connecting the Dots
- 24:58Adding Images
- 27:07Running on a GPU
- 32:45Wrapup
related talks
27mHarnessing Kubernetes for Scalable AI/ML Workloads
32mGPU Machine Learning From Laptop to Cloud
37mBuilding Distributed TensorFlow Using Both GPU and CPU on Kubernetes [I]
01h 26mTutorial: Introduction to Kubeflow Pipelines
38mKubernetes for AI Workloads – What Works and What… Doesn’t
25mMaking the Most Out of Your Hardware Accelerators in a Kubernetes Cluster
34mGPU as a Service Over K8s: Drive Productivity and Increase Utilization
29mKubeflow: TensorFlow on Kubernetes