Preparing Data for LLM pretraining using open source Data Prep Kit
summary
Starts from Common Crawl and other sources and shows actual steps: PDF→par extraction, paragraph chunking, exact and fuzzy deduplication that drops ~60–70% duplicates, language ID (11 languages), LLM as judge scoring for hate/abuse, code‑quality checks, and embeddings with sentence‑transformers saved to a vector DB. Shows config knobs, Ray/Spark scaling, checkpointing, and that 28.7 TB of input falls to ~8 TB after filtering so viewers can judge tool coverage and measurable impact.
AI-written summary · about
chapters (24)
- 0:00LLM data preparation intro
- 1:07Why data quality matters
- 3:58Data as crude oil
- 7:13Internet-sourced data challenges
- 12:33Data quality assessment
- 16:33IBM Granite 3.0 pipeline
- 21:23Introducing Data Prep Kit
- 26:00Data Prep Kit features
- 33:29Lab setup first transform
- 42:09PDF to Parquet demo
- 50:12Chunking documents
- 1:00:12Dirty PDFs handling
- 1:03:04Chunk array transform
- 1:08:26Adding document IDs
- 1:10:15Exact duplicate removal
- 1:12:50Fuzzy duplicate detection
- 1:15:07Generating embeddings
- 1:16:53Code data prep intro
- 1:18:40Preparing GitHub code
- 1:21:11Code dataset processing
- 1:24:12Extending with custom transforms
- 1:26:48Code quality filtering
- 1:28:33Semantic alignment
- 1:29:20Summary and contribution
related talks
28mThe Data That Shapes Foundational LLMs
22mCase Study in Bridging Production Software and Data Practices for LLM Model Training
33mDeep research at scale
30mLearning to wield large language models
42mKeynote: Foundation to Build LLMs
40mInstructLab: Applying Open Source Methods to Building and Trai...
15mKeynote: Common Corpus: Opening Data for Building Open Source LLMs
29mLocal LLM fine-tuning: a practical example