Operating Deep Learning Pipelines Anywhere Using Kubeflow
summary
Shows how to assemble a production-grade ML pipeline on Kubernetes using Kubeflow, with notebook-driven model engineering, distributed training and autoscaling for TensorFlow clusters, and reproducible deployments via pipelines or CI/CD (Jenkins). Data preparation uses Apache Spark (or Flink), storage uses HDFS/Cassandra, streaming uses Apache Kafka, and model storage/serving integrates model registries and serving layers. Covers hyperparameter tuning, model optimization for CPU/GPU/mobile, multi-tenant resource isolation, and the need for a shared metadata/feature catalog for reproducibility.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 13:10The Advanced Pipeline
- 21:01Challenges
- 22:46Data Quality
- 24:44Division of Labor
- 26:24Data Science vs Software Engineering
- 26:57Data Science Engineering Principles
- 27:19Ian Good
- 27:38Requirements engineering
- 29:24Reproducible builds
- 31:04mlflow
- 31:44data scientist
- 32:38Jupiter
- 33:20Feature Catalog
- 34:41Model Libraries
- 35:22Model Optimization
- 36:26Resource Service Management
- 37:08Wishlist
related talks
54mScaling Machine Learning Pipelines in Cloud
42mDeep learning beyond the learning
51mMachine Learning using Kubeflow and Kubernetes
46mDeploying scalable Machine Learning models with Kubernetes
05mLightning Talk: Build a Reproducible ML Workflow with Kubeflow Pipelines
35mBringing Your Data Pipeline into The Machine Learning Era
14mKubeflow: Where ML Meets Kubernetes
39mDeploy your ML Workloads on Kubernetes with Kubeflow