Managing Thousands of Automatic Machine Learning Experiments with...
summary
Katib plus Argo Workflows runs hyperparameter tuning where each trial can spawn a full Argo workflow, letting preprocessing be cached and skipped via Argo caching stored in Kubernetes configmaps. The system supports multi‑objective tuning, early stopping, custom suggestion algorithms and runs parallel trials (demo uses 2 parallel, 5 max) as Katib child template trials. The integration is production‑focused, orchestrates CRDs for distributed training, and demonstrably reuses cached outputs to save compute.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:28KDP Overview
- 2:27KDP Architecture
- 3:43Why Algo Workflows
- 5:35Memorization Cache
- 6:49Template Spec
- 7:41Example Workflow
- 9:33Entry Point
- 10:33MultiObjective Optimization
- 11:42Implementation
- 12:33Demo
- 19:28Community
- 20:43QA
related talks
22mBuilding AutoML Pipelines With Argo Workflows and Katib
35mHyperparameter Tuning Using Kubeflow
25mPanel: Argo for ML: Achieving Scalability and User Experience
10mScaling Time Series Analysis With Argo Workflows: Patterns and Practi...
54mScaling Machine Learning Pipelines in Cloud
27mEmpowering Autonomy: BYD's Journey Taming Million-Task Scale With Argo...
25mTraining and Ensuring Reliability of ML Models at Wolt: The Power of ArgoCD, Flyte, and Argo Workflows
07mArgo in an AI-Native World: Effective In-Cluster Machine Learning and A...