Olmo-Thinking: Training a Fully Open Reasoning Model
summary
7B Olmo-Thinking claims parity with Qwen 3 by combining midtraining, targeted SFT distillation from high-quality “thinking” data, and reinforcement learning with verifiable rewards; small-model distillation works well and direct preference optimization yields ~+2–3 points when chosen answers come from Qwen 3 32B and rejects from a ~0.6B model. A key engineering win is GQA gives ~8x memory reduction in attention, enabling RL at 7B that would otherwise cost like a 32B setup. RL runs use continuous off‑policy generation, recomputed logprobs or importance sampling, and on‑the‑fly weight updates.
AI-written summary · about
chapters (10)
- 0:00Open reasoning models intro
- 1:04RL limits in language modeling
- 3:52Need for fully open reasoning
- 4:34Reasoning model architecture
- 6:00Pre-training data selection
- 7:43Long context challenges
- 9:07Post-training for reasoning
- 11:52RL fine-tuning challenges
- 13:17GPU utilization in RL
- 14:42Next steps for scaling
related talks
32mScaling Test-time Inference Compute & Rise of Reasoning Models
25mThe Future of Qwen: A Generalist Agent Model
15mAgentic Reasonings with Reinforcement Learning
29mThe Evolution of LLMs: From Imitation to Reasoning
24mThe Rise of Reasoning Models
01h 00mDeepSeek-R1 Thoughtology: ＜Thinking＞ about LLM Reasoning
19mTraining Agentic Reasoners
39mUnlocking reasoning in Large language models