Which Is To Be Master? Understanding LLM Tokenization
summary
Compares concrete tokenizers—Byte Pair Encoding, WordPiece, and unigram tokenizer—and trains tokenizers from corpora, then runs HMM surrogates to measure impact: small vocabularies produce garbage, larger ones recover recognizable text; vocabulary size dominates memory (V ≈ 20× embedding dim) and fertility (tokens/characters) tracks compression. Cites a paper showing an optimal vocabulary region, finds many popular models off that optimum, and reports ~70% retrieval for missing-word embeddings. Code and experiments are on GitHub.
AI-written summary · about
chapters (11)
- 0:00LLM tokenization basics
- 1:03How LLMs tokenize text
- 3:55What tokenizers do
- 7:09Tokenizer algorithms
- 12:30Special tokens role
- 13:38Why tokenization cannot be changed easily
- 15:25Tokenization cost vs fidelity
- 17:13Vocabulary size trade-offs
- 18:15Tokenization performance impact
- 22:26Tokens vs LLMs
- 23:27Q&A
related talks
46mThings you never dared to ask about LLMs
46mUnder the Hood of LLMs — all those questions you never dared ask about them
37mExpanding the Capabilities of LLMs Through Community Collaboration
34mTerrible tokenizer troubles in large language models
49mBen Hall & Barbara Fusinska
30mWhere there's a will, there's a way: The mad genius of LLMs as a classifiers
22mSmarter word generation in LLMs: Guiding sampling through output probability distributions- 37mExpanding the Capabilities of LLMs Through Community Collaboration