Training an LLM from Scratch, Locally
summary
Claims that a usable LLM can be trained locally are backed by a runnable repo: PyTorch code split into model.py, train.py and generate.py, runnable on a laptop with ~16 GB or Google Colab T4. The workshop actually builds a GPT‑2–style model with a 65‑token char tokenizer, block size 256, 6 layers, 6 heads, 384 embeddings (~1.8M parameters), trains ~5,000 steps with 100‑step warmup and batch 64, watches loss fall (decent outputs near loss ~1.0, overfit around 2,400), and generates Shakespeare‑like verses.
AI-written summary · about
chapters (18)
- 0:00Introduction and background of the speaker
- 1:21Overview of the workshop objectives
- 3:12Inspiration from Andre Karpathy's NanoGPT
- 4:37The four fundamental building blocks of an LLM
- 7:08Prerequisites and setup tools (UV, Python, hardware requirements)
- 9:06Part 1: The Tokenizer (character-level tokenization explained)
- 24:29Model architecture and parameters (vocab size, layers, embeddings)
- 30:13The GPT class structure and transformer blocks
- 37:52Parameter count and model sizing
- 40:54The training loop: objectives and next-token prediction
- 44:44Optimization and learning rate strategies (warm-up and cosine decay)
- 47:56Validation and monitoring loss
- 53:07Part 3: Text generation and inference strategies
- 56:30Putting it all together (project file structure)
- 58:46Monitoring training and debugging common issues
- 1:00:27Workshop challenge and competition details
- 1:05:24Q&A: Differences between base models and reasoning models
- 1:11:31Q&A: Applying these concepts to audio and multimodal models
related talks
44mHow to build your own Tiny Language Model from scratch
30mLearning to wield large language models
29mLocal LLM fine-tuning: a practical example
33mAccelerating the LLM Lifecycle on the Cloud
24mHarnessing the Power of LLMs in Advanced Analytics
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
43mCreating Your Own Copy with a Fine-Tuned LLM
02h 52mLow Level Technicals of LLMs