Automate the boring ML stuff with pipelines
summary
A reproducible ML pipeline that validates incoming data against a generated schema and stops training on mismatched batches, runs feature engineering with TensorFlow Transform, trains distributed models with a mirrored strategy, and evaluates new models against prior versions using TFMA rules (example gate: require precision +2% to deploy). Orchestration runs on Kubeflow Pipelines or Airflow, scheduling weekly or on new-batch triggers, pushing model plus preprocessing via a pusher component and recording full lineage and metadata for audit and rollback.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:46What is in a production ML system?
- 3:07Most models don't get deployed
- 4:24What is a ML Model Lifecycle?
- 5:04TensorFlow Extended
- 5:51TFX Components
- 6:18What is TFX and why do we use it?
- 6:40TFX Pipeline Orchestration
- 7:28Data Ingestion Define versions and spans
- 9:24Data Transformation
- 10:13Model Training Very similar to "manual" training
- 11:21Model Analysis • Define criteria for model performance
- 12:57Fairness in an ML pipeline
- 13:25TensorFlow Model Analysis
- 13:55Model Deployment Push models to file system
- 14:15Execute your TFX Pipeline
- 14:26Execute the Pipeline
- 15:24Kubeflow Pipelines
- 15:38Scheduled Model Updates
- 15:53Model Lineage Tracking
related talks
24mAutomating ML Pipelines With Kubernetes And Airflow
05mLightning Talk: Build a Reproducible ML Workflow with Kubeflow Pipelines
29mML Engineering Kubeflow
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
35mOrganizing data engineering around Kubeflow
39mDeploy your ML Workloads on Kubernetes with Kubeflow
45mAutomate your ML workflows with pipelines
43mMachine Learning Engineering principles with Python and MLFlow