Structural Transfer Learning: Exploring Neural Models and Language Structure Through...
summary
Pretraining small transformer models (GPT-2 small) on synthetic symbolic data—nesting parentheses, crossing dependencies, a regular repeating control and random noise—then fine-tuning on Wikipedia (~100 million tokens, Basque ≈50M) measures which structural biases speed language learning. Results show nesting improves perplexity versus random, crossing dependencies often match or beat nesting, small amounts of crossing (1–10%) degrade constituency gradually, and Zipfian vocabularies outperform uniform distributions across English, Japanese and Basque.
AI-written summary · about
chapters (16)
- 0:00Introduction to study
- 0:37Transfer learning methodology
- 2:27Analytical testbeds
- 4:41Role of structure and NLP
- 7:27Defining linguistic structures
- 11:04Recursion and nesting
- 14:15Crossing dependencies
- 16:18Vocabulary distribution
- 18:47Experimental design
- 22:41Synthetic language controls
- 26:35Testing structure hypotheses
- 33:08Results and findings
- 38:48Comparing structure types
- 46:24Mixed structure experiments
- 50:34Vocabulary and conclusions
- 55:32Q&A and future directions
related talks
25mGenerative Text Modelling: Scratching the surface
44mTransfer learning in NLP
23mUnveiling the Journey of NLP
01h 16mGeneralization in natural language processing
01h 26mFrom RNNs to Transformers: A Journey through Language Modeling
50mKnowledge is embedded in language neural networks but can they reason?
43mThe Evolution of Domain Transfer in the Era of Pre-trained Language Models
01h 38mTraining Large Language Models: Practices and Research Questions