Fine tune nanoGPT for Microsoft Azure
summary
Demonstrates fine-tuning nanoGPT on a three-node Azure cluster built from D32 v5 VMs (one master + two workers), using the OpenWebText dataset, Hugging Face Accelerate, and Intel tooling to turn a single-node PyTorch run into distributed training. Uses Intel Extension for PyTorch and oneCCL with mpirun/Intel MPI, runs 10 iterations with batch size 32, saves checkpoints and snapshot.pt, and successfully generates samples from the resulting model.
AI-written summary · about
chapters (7)
- 0:00Intel optimized cloud modules
- 1:58Fine-tuning NanoGPT project
- 3:32Single-node setup and training
- 7:23Single-node configuration
- 9:27Distributed environment setup
- 14:40Distributed training execution
- 16:46Model inference and conclusion
related talks
28mTraining Large Language Models on Kubernetes
28mHarnessing generative AI with NVIDIA AI and Microsoft Azure
41mGPU Training Made Easy-Pre-train & fine-tune NLP models
01h 02mWorkshop: LLMs Fine Tuning and Inferencing Using ON...
33mBuild Better AI with First-Party Data and Fine-Tuning
36mAccelerating Machine Learning at Scale with
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective