Training Large Language Models in the Open
summary
Claims concrete: 176B‑parameter model trains on roughly 400 GPUs on the Jean supercomputer, BigScience builds a 350B‑token dataset, and the code effort crawls ~140M GitHub repos into a 3–10 TB corpus. SantaCoder runs 1B parameters on 250B tokens for about $8k; StarCoder scales to ~50B parameters and up to 4T tokens. Governance tools include data inspection opt out and a membership test that flags exact-token training matches.
AI-written summary · about
chapters (9)
- 0:00Open LLM introduction
- 3:53Open models in AI
- 7:45Closed vs open models
- 10:14BigScience project
- 13:463D parallelism training
- 16:55Stack dataset for code
- 20:07StarCoder evolution
- 21:35Code model evaluation
- 23:23Open models impact
related talks
31mBuilding Large-scale Localized Language Models
28mFine-Tuning Small Language Models with Python
01h 38mTraining Large Language Models: Practices and Research Questions
18mImproving open language models at a practical size
30mLarge Language Models and Prompt Engineering for Developers
49mBen Hall & Barbara Fusinska
19mSystem for Scaling Large Language Models
39mCollect Data to Fine-Tune Your Large Language Model