From Von Neumann to Transformers: An Infra Engineer's Guide to What's Actually Inside an LLM.
summary
A 110M-parameter GPT-2 clone is trained from scratch on ~1GB of German text using a 32k-token vocabulary and 768-dim embeddings, producing German-like output within minutes as loss starts to fall. Training runs on two H100 GPUs at ~500–600k tokens/s, the model uses 12 transformer layers with 8 heads each (96 heads total), and relies on embedding matrices, dot‑product logits, softmax and attention plus small feed‑forward networks — an explicit analogue-style compute architecture rather than an oracle.
AI-written summary · about
chapters (19)
- 0:00Welcome and introduction
- 1:08Von Neumann vs modern AI
- 3:38Softmax and real-number computers
- 5:25From tokens to vectors
- 8:38Embedding matrices explained
- 11:47Vector arithmetic and semantics
- 15:16Neural networks as circuits
- 17:41Training an LLM step-by-step
- 19:49GPT architecture breakdown
- 21:58From hidden states to tokens
- 23:41Auto-regressive generation
- 25:07Debugging with logit lenses
- 26:52Transformer blocks and attention
- 28:38How attention works
- 31:53KV caching and real numbers
- 32:14LLMs as analog computers
- 33:58Further learning resources
- 34:41Key takeaways and inspiration
- 35:47Q&A on training and unpredictability
related talks
01h 00mInside GPT – Large Language Models Demystified
47mInside GPT – Large Language Models Demystified
19mSystem for Scaling Large Language Models
43mBeyond the Hype: A Realistic Look at Large Language Models
01h 16mHow Do Transformers Work?
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
41mInside GPT – The Maths Behind the Magic
01h 26mTransformers Inside Out (Parts 1 & 2)