Case Study in Bridging Production Software and Data Practices for LLM Model Training
summary
Snowflake hosts the entire LLM data pipeline: data lands as tens of terabytes from Common Crawl and Hugging Face, is split into a file metadata/file contents split to save storage, and a global dedup uses content hashes (about 20% of files duplicate). Quality signals include a percent HTML visible UDTF (Python + BeautifulSoup) and an is noisy XML flag, with policy enforced by views, provenance via migration-driven exports and output cards, Streamlit apps for human labels, diff tools, and hundreds of ablation runs; Arctic ships from empty dir to model in under five months.
AI-written summary · about
chapters (6)
- 0:00Arctic LLM training pipeline
- 4:22Code dataset with Snowflake
- 12:29Data lineage and policy enforcement
- 14:56Data quality iteration tools
- 19:27Evaluating data filtering
- 20:30Future directions for LLM training
related talks
40mPreparing data for LLM training with Data Prep Kit
49mLLM inference and fine-tuning on secure enterprise data
01h 30mPreparing Data for LLM pretraining using open source Data Prep Kit
29mData management for enterprise-grade foundation models.
37mAdrian Brudaru | Data Track | Compass AI & Data Summit 2025
28mThe Data That Shapes Foundational LLMs
28mNavigating LLM deployment: Tips, tricks, and techniques
29mBuilding End to End Gen AI Powered Analytics: From Data to Production