Building Cross-Cloud ML Pipelines with Kubeflow with Spark & Tensorflow
summary
Shows how Kubeflow, Spark and TensorFlow form a repeatable cross‑cloud pipeline: use JupyterHub for exploration, Spark for distributed feature prep, then TensorFlow training and serving with TFJob + TFServing, and add a final pipeline stage to upload the model to cloud object storage so another cluster can download and serve it. Pipelines are thin pipelines=Argo wrappers. Warns Spark Python path copy is slow unless using pandas UDF/Arrow (real speedup ~10–20%), and recommends baking Python deps into containers on Kubernetes.
AI-written summary · about
chapters (18)
- 0:00Kubeflow book!
- 1:45Holden About Me Slides
- 2:59Trevor Grant
- 6:31Components Buffet
- 8:07Model Serving
- 8:30How does this fit in to the Ecosystem
- 11:10Getting the chef's recommend pairing
- 13:39What are those pipelines?
- 14:42Using Spark + TF together in KF
- 16:52A quick detour into PySpark's internals
- 20:30Arrow (a poorly drawn big data view)
- 21:05How to swap to using Arrow
- 22:08How PySpark Dependencies work
- 24:30What do the rest of the systems do?
- 26:17Cross Cloud OR The part of the talk where trevo shoot's his employer in the back.
- 28:04Clouds are becoming Commoditized
- 32:12Train on cheapest cloud of the day
- 34:26Questions? Half-baked Demo?
related talks
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
21mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
33mDeploying and Scaling Spark ML and Tensorflow AI Models
54mScaling Machine Learning Pipelines in Cloud
29mKubeflow: TensorFlow on Kubernetes
05mLightning Talk: Build a Reproducible ML Workflow with Kubeflow Pipelines