Building Large-scale Localized Language Models
summary
A project trains 1.7 billion parameter models for Spanish and French from OSCAR, cleans and deduplicates 282GB to 117GB, tokenizes in ~45 minutes, and converts to mapped binaries. Training uses Megatron-LM with tensor parallel=2 on DGX‑1 V100 nodes, costing about 5000 V100 GPU hours per model. Models generate coherent language, perform few‑shot tasks well, and deploy via conversion to FasterTransformer and Triton; quantization and pruning cut memory footprints dramatically.
AI-written summary · about
chapters (14)
- 0:00Localized language models intro
- 1:28Training Spanish/French locally
- 3:34Transformers revolution
- 8:31Data prep steps
- 13:23Training 1.7B model
- 16:35Distributed training strategies
- 18:41Evaluating text examples
- 19:23Downstream NLP tasks
- 21:53Prompt learning techniques
- 22:56Deploying to production
- 25:29Inference efficiency
- 27:37Live inference demo
- 28:19Key takeaways & resources
- 29:44Q&A on Triton Server
related talks
25mTraining Large Language Models in the Open
28mFine-Tuning Small Language Models with Python
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
30mLearning to wield large language models
42mPost-Training Small Language Models: the adventures of a practitioner
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
44mHow to build your own Tiny Language Model from scratch
38mUnraveling Multimodality with Large Language Models