Generative AI Model Data Pre-Training on Kubernetes: A Use Case St...
Talkat KubeCon + CloudNativeCon Europe 2025 · on Thursday, 2025-04-03· at 16:00· Level 1 | Hall Entrance N11·
summary
Demonstrates a production LLM data stack that uses Kubeflow Pipelines to orchestrate Ray jobs via KubeRay and a Data Prep Kit (DPK) for reusable transforms. Shows a fuzzy deduplication run over 8.5 billion documents (23 TB compressed) that cuts documents and storage by ~33–40% using a Ray cluster sized at ~7,500 CPU cores and 56 TB RAM over ~40 hours. Describes KFP v1 workarounds for nested pipelines, KFP v2 native nesting, and DPK as an open source, multi-runtime toolkit.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:03Data pre-processing workflows
- 3:53Scaling with Ray and KubeRay
- 7:30Kubeflow Pipelines orchestration
- 11:19Modular pipelines in Kubeflow
- 13:44Live demo: multi-step pipelines
- 19:22Data Prep Kit project
- 22:38Resources and community
- 23:22Q&A
related talks
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case Study
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case S...
34mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case Study
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
51mMachine Learning using Kubeflow and Kubernetes
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
33mCompliant Data Management and Machine Learning on Kubernetes
16mScaling Massive Transformer Training with KubeRay