Preparing Data for LLM Applications Using Data Prep Kit
summary
Open-source Data Prep Kit shows a concrete pipeline: PDF→chunks→dedupe→embeddings using Python tools (Colab demo, DocLing PDF extractor, sentence-transformer embeddings) and produces vector-ready chunks that drop 11 paragraph chunks to 9 after dedupe in the demo. It removes PII, detects language and malware, and creates embeddings. The code runs locally or scales via uses Ray or Spark to scale and is proven — IBM used it to prepare data for Granite trained on multiple terabytes across thousands of machines.
AI-written summary · about
chapters (13)
- 0:00Workshop welcome
- 2:28Speaker and toolkit
- 5:42Data Prep Kit overview
- 19:20Understanding chunking for LLM applications
- 23:15Setting up the environment for hands-on demo
- 26:49PDF processing demo
- 32:16PDF text extraction
- 43:25Chunk splitting
- 50:54Eliminating duplicate documents
- 58:41Fuzzy deduplication
- 1:01:55Vectorizing for search
- 1:12:35Workshop recap and scaling
- 1:14:45Resources and closing
related talks
01h 19mLearn to Unlock Document Intelligence with Open-Source AI
36mTechniques to avoid common LLM pitfalls
33mDesigning LLM Applications for Massive Datasets
18mLLMs and GenAI: A Guide to Building Your Own RAG Model for Document Interaction
25mUsing Large Language Models to Improve Data Loss Prevention in Organizations
01h 19mBuilding an LLM on your Laptop to...
30mFrom Unstructured Docs to Graph Intelligence: A Framework for Building Graph+Embedding Pipelines
01h 00mKeynote: Building and Productionizing RAG