Fine-tune and deploy LLMs from Hugging Face (AIM304)
summary
Shows how to fine-tune a Llama 370B checkpoint using AWS Trainium with tensor and pipeline parallelism plus ZeRO stage 1 across multiple nodes, then compile and deploy the consolidated checkpoint to Inferentia2 via the Optimum Neuron library. Claims: up to 50% training cost savings on Trainium, up to 4× throughput and 10× lower latency on Inferentia2, and 4.7× cost reduction for inference. Demonstration builds a chat bot, compiles artifacts with caching, and generates responses successfully.
AI-written summary · about
chapters (10)
- 0:00Introduction and agenda
- 1:32LLM customization techniques
- 3:49Benefits of smaller models
- 5:59AWS Trainium and Inferentia
- 12:07Performance and deployment
- 17:35Software and SDK overview
- 22:10Hugging Face and Optimum
- 26:22Optimum Neuron capabilities
- 30:10Fine-tuning and deployment
- 39:20Conclusion
related talks
27mFoundation Models: Blueprints for success
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
30mUnderstanding the end-to-end LLM training and inference pipeline
28mNavigating LLM deployment: Tips, tricks, and techniques
23mCost-Effective RL Post-Training with Ray and AWS Trainium
02h 52mLow Level Technicals of LLMs
29mLocal LLM fine-tuning: a practical example