~/talk

Generative AI Model Data Pre-Training on Kubernetes: A Use Case Study

on YouTube 22m
Talkat DevConf.CZ 2025 · 2025-06-12..06-14· [machine-learning, kubernetes]

speaker

name:Anish Asthana
bluesky:anishasthana.bsky.social

resources

video:https://youtu.be/iyu_yggCNmk
slides:none on file

summary

Demonstrates a production pipeline using Kubeflow Pipelines (KFP), Ray via KubeRay, and the open Data Prep Kit (DPK) to preprocess PetaByte-scale LLM data with reproducible, nested pipelines and per-step Ray clusters. A fuzzy deduplication run on 250,000 parquet files (23 TB compressed, ~8.5 billion docs) reduces to 5.6 billion documents and ~14 TB stored, running on 7,500 CPU cores and 56 TB RAM over 40 hours; checkpointing and exit handlers prevent losing days of work.

AI-written summary · about

chapters (9)

  1. 0:00Introduction
  2. 1:03Data processing workflows
  3. 3:33Scaling data prep with Ray on K8s
  4. 6:45Reproducibility with Kubeflow Pipelines
  5. 10:18Case study: data pre-processing pipeline
  6. 13:29Live demo: super pipeline run
  7. 16:17Results: largest data prep job
  8. 17:41Consistent development with Data Prep Kit
  9. 20:58Q&A

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.