Kubeflow Trainer Observability: Real-Time Progress Tracking for Reliab...
summary
Adds real-time training metrics into TrainJob status on the Kubernetes API so Katib can prune trials and save GPU-hours instead of parsing logs or running MLflow. Implements a push-based status server, Hugging Face Transformers callback and an SDK utility, injects endpoint and auth env vars into pods, requires no extra RBAC and no sidecars. Demonstrated on hyperparameter search scenarios (50 configs, 16 GPUs × 8h → 6,400 GPU-hours) and claims 5–10x cost reduction visible with a simple kubectl get query.
AI-written summary · about
chapters (29)
- 0:00Kubeflow Trainer Observability intro
- 1:03What is Kubeflow Training
- 2:29Hidden costs of distributed ML
- 3:53Observability gaps are costly
- 5:18Current training job monitoring
- 7:03Real-time progress tracking in Kubeflow
- 8:07Design decisions behind solution
- 9:09Push-based updates vs scraping
- 10:11Security and permission considerations
- 10:52Avoiding sidecars for memory
- 11:13Key components of solution
- 12:17Zero-code Hugging Face integration
- 12:39Live demo progress tracking
- 15:28Demo setup and job submission
- 16:09Querying metrics via Kubernetes API
- 16:51Custom training with SDK utility
- 17:57Zero infrastructure changes benefits
- 18:18Future roadmap ideas
- 19:00Hyperparameter optimization with Katib
- 20:23Trial pruning and early stopping
- 21:05Kubeflow training UI metrics
- 21:48Elastic training GPU management
- 22:10Job preemption policies
- 22:30Kubeflow MCP server
- 22:51Setting deadlines auto-extending
- 23:32Release details next steps
- 24:13Q&A dataset preprocessing
- 25:16Q&A MLflow integration
- 26:24Q&A SDK design choices
related talks
29mKubeflow: TensorFlow on Kubernetes
31mKubeflow Intro – Michał Jastrzębski & Ala Raddaoui, Intel
21mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
14mKubeflow: Where ML Meets Kubernetes
36mStreamlining ML Workflows With the Unified Kubeflow SDK
42mKubeflow From the End User’s Perspective: The Good, The Bad, and The Ugly
36mKubeflow: Tensorflow on Kubernetes
51mMachine Learning using Kubeflow and Kubernetes