Data management for enterprise-grade foundation models.
summary
IBM describes an enterprise pipeline that ingests web snapshots (eg Common Crawl ~1 TB per snapshot, 30–40 snapshots), performs large-scale deduplication across snapshots, extracts text, runs document-quality classifiers, hate/abuse detection and PII masking, tokenizes and feeds models (notes GPT‑4 used ~13 trillion tokens). Storage moves from raw object buckets to a lakehouse using Iceberg table format for tagging/time‑travel and fewer copies. Ray/Spark run preprocessing, Trino/Presto power queries; lineage, access control and Ray–Iceberg write integration remain work in progress.
AI-written summary · about
chapters (11)
- 0:00Introduction to foundation models
- 2:20Foundation model components
- 5:14Enterprise-grade model approach
- 9:08Data acquisition and governance
- 10:16Data preprocessing pipeline
- 14:15Data provenance and lineage
- 16:45Data Lake challenges
- 19:14Transition to data Lakehouse
- 22:35Current challenges and opportunities
- 25:23Risk mitigation beyond data
- 27:14Summary and IBM’s role
related talks
22mCase Study in Bridging Production Software and Data Practices for LLM Model Training
34mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case Study
37mData Swamp to Data Swimming Pool: Cleaning up the Mess
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications
29mLakeside AI: Bringing Models to Data with Open Lakehouses & Agents)
35mHigh-performance data foundation for AI success
31mBuilding a Data Infrastructure for AI/ML
50mDesigning a scalable data platform based on Apache Spark/Iceberg