A Very Small Language Model
summary
A working tiny language model gets built in about an hour using a JavaScript Markov chain implementation: a tokenizer generator, a Chain class that records follower counts and simple weighted sampling. Trained on a personal Mastodon dump (~46,576 words corpus) plus novels, it yields plausible single sentences; adding start/end sentence markers improves coherence, widening the n‑gram window to two helps, but widening further collapses into regurgitating training text. Runtime training, POS tags and recency weighting are viable extensions.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 10:10Next word prediction limits
- 13:20Markov chains basics
- 22:32Text generator in JS
- 30:12Training with text corpus
- 38:09Output with sentence markers
- 43:49Advanced text enhancements
- 53:04Combining text sources
- 58:42Window size for coherence
- 1:00:08Small systems and risks
related talks
44mHow to build your own Tiny Language Model from scratch
16mExploring the world of Small Language Models
43mComedy Writing With Small Generative Models
49mBen Hall & Barbara Fusinska
18mHow to Make Small Language Models Work
01h 24mIntroduction to Language Modeling
34mTerrible tokenizer troubles in large language models
25mTraining Large Language Models in the Open