Generative AI Model Data Pre-Training on Kubernetes: A Use Case Study
summary
Shows a production pipeline using Kubeflow Pipelines + KubeRay to spin up Ray clusters, run deduplication, tokenization and content classification, then tear down resources; a fuzzy dedupe job shrinks 42 TB in-memory input to 14 TB on disk (5.6B docs) using 7,500 CPUs and 56 TB RAM in ~40 hours. Claims 70–80% of raw web data is dropped before pretraining, and reducing FineWeb from 15T to 10T tokens improves downstream model metrics.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:05Data quality for LLMs
- 3:37Data oil analogy
- 6:51Data pipeline challenges
- 11:03LLM data engineering challenges
- 15:19Scalable data processing tools
- 19:15Kubernetes and Qflow
- 24:08Data Prep Kit overview
- 28:53Open-source future
- 29:41Q&A
related talks
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case Study
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case S...
27mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case St...
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
46mAccelerating Machine Learning DevOps with Kubeflow
36mNavigate the future of AI: Deploy generative models on Amazon EKS
16mScaling Massive Transformer Training with KubeRay
01h 26mTutorial: Introduction to Kubeflow Pipelines