Preparing data for LLM training with Data Prep Kit
summary
Claims a production pipeline and an open source tool called Data Prep Kit (DPK) that actually turns Common Crawl into training-ready Parquet, using PDF/HTML extractors, tokenization, language ID, and fuzzy dedup with MinHash. Reports cutting 28.7 TB to ~13.8 TB then to ~8 TB after quality filters, runs scaling on Ray/Spark, and performs 150 ablation studies to choose filters. The pipeline backs IBM Granite 3.0 and is presented as a working, extensible system.
AI-written summary · about
chapters (16)
- 0:00Data preparation for LLMs
- 1:30Data quality for LLM training
- 3:00Raw internet data challenges
- 6:58Internet's role in LLM data
- 8:27Bias toxicity copyright in data
- 10:12Data quality filters techniques
- 12:26Clean data for fine-tuning
- 15:41Ablation studies for filtering
- 19:56Balancing content filtering strictness
- 21:24IBM Granite model data journey
- 24:38Data Prep Kit introduction
- 28:20Scaling with parallelization
- 31:10Custom pipelines in Data Prep Kit
- 33:43Use cases beyond pre-training
- 34:50Open source resources
- 36:41Q&A
related talks
38mData Prep Kit: A Comprehensive Cloud-Native Toolkit for Scalable Da...
22mCase Study in Bridging Production Software and Data Practices for LLM Model Training
27mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case St...
45mWorkshop 3: Data Engineering - A Roadmap... by Ramon Perez [Part 3]
27mData Prep for Graphs
01h 00mComplex Data Ingestion with Open Source AI
28mThe Data That Shapes Foundational LLMs
33mDeep research at scale