No Data? No Problem: Zero-Data Model Training Foundational Models
summary
Claim: generate training data from a large foundational model and fine‑tune a smaller model with no human labels. Implementation uses Llama 8B quantized as the data generator, TinyLlama 1.1B as the target, and PEFT/LoRA to adapt it. Demo produces 3,000 synthetic examples, fine‑tunes in ~15 minutes, saves model to a distributed volume, and deploys via the Covalent/Calent Python platform using H100 and L40 GPUs. The end‑to‑end notebook builds and runs the pipeline and the deployed classifier mostly works though some edge cases mislabel.
AI-written summary · about
chapters (11)
- 0:00Introduction to the session
- 1:33HPC vs. general computing
- 6:03Covalent platform overview
- 8:51Workflow and design concepts
- 11:58Notebook setup and configuration
- 14:43Defining services and endpoints
- 23:21Deployment and infrastructure
- 29:02Agent orchestration workflow
- 36:47Live demo and model testing
- 43:43Scaling and improvements
- 46:43Q&A and final remarks
related talks
22mSpecializing Small Language Models with less data
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...
28m🧼 From GPU-poor to data-rich: data quality practices for LLM fine-tuning
30m🤯 No data? No problem! Synthetic data to the rescue
31mBuilding LLaMA: Meta's Director of GenAI
25mFine-Tuning Large Language Models With Declarative ML Orchestration
31mReliable Data for Large ML Models: Principles and Practices
28mFine-Tuning Small Language Models with Python