Controlling distribution shifts in language models: a data-centric approach.
summary
Two complementary, data‑centric methods claim practical control over pretraining gaps. First, a rank‑based single‑index estimator uses log‑loss differences from 90 public models to compute domain weights, then sorts domains by a gamma score and samples until the token budget, validating on eight benchmarks with tiny 160M parameter models and matching a heavily handcrafted Datacomp‑LM pipeline. Second, a synthesis pipeline samples entity subgraphs, prompts a large LM to expand them, and continues pretraining on only 1.3M tokens, boosting closed‑book QA to ~56% with far fewer hallucinations than raw CPT or paraphrase baselines.
AI-written summary · about
chapters (8)
- 0:00Language model challenges
- 3:51The data selection problem
- 10:38Algorithmic data selection
- 20:01Single index model approach
- 25:59Evaluation and validation
- 33:12Data synthesis for adaptation
- 42:04Entity-centric augmentation
- 48:03Results and conclusion
related talks
01h 38mTraining Large Language Models: Practices and Research Questions
28mThe Data That Shapes Foundational LLMs
46mUnder the Hood of LLMs — all those questions you never dared ask about them
41mControlling Dependency on Third‑Party Large Language Models
43mThe Evolution of Domain Transfer in the Era of Pre-trained Language Models
51mTowards a Pragmatic Understanding of Large Language Models
22mSpecializing Small Language Models with less data
58mFederated Open Language Models: Training LMs on Distributed Data