RFT, DPO, SFT: Fine-tuning with OpenAI
summary
Covers concrete fine-tuning recipes and outcomes on the OpenAI platform: using the Chat Completions API to SFT function-calling with 120 functions, generating low- to mid‑thousands of synthetic examples (20–200 per function) and distilling GPT‑4 behavior into GPT‑3.5 to reach within ~2% of GPT‑4 accuracy at far lower latency. Rules of thumb: 50–100 examples shows signs of life, 500+ for production; RFT needs ~40–80 very high‑signal examples and lifts reasoning tasks (chemical donor/acceptor accuracy ~65% → ~80%); DPO suits tone/AB signals; noisy subjective labels (email case) break RFT.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:14LLM Optimization
- 4:37Fine-tuning Types
- 5:53Fine-tuning Techniques
- 12:37SFT Success Story
- 24:45SFT Demo
- 34:03SFT Results & Tips
- 41:22DPO Demo
- 45:28RFT Introduction
- 52:05RFT Example
- 55:26Q&A Session
- 1:00:58RFT Deep Dive
- 1:13:06Retrieval & Tuning
- 1:16:18Email Classifier Failure
- 1:19:36Prompt Tuning Experiment
- 1:24:52Debugging Experiment
- 1:33:22Final Q&A
related talks
19mIntro to Fine-tuning
43mState of GPT
21mBeat GPT 4 and Build the Best AI For You: Mobilizing First-Party Data with Fine-Tuning
19mBeat GPT 4 & Build the Best AI For You
30mHow to train GPT-3 with your own dataset
28m🧼 From GPU-poor to data-rich: data quality practices for LLM fine-tuning
21mBeat GPT 4 and Build the Best AI For You: Mobilizing First-Party Data with Fine-Tuning
38mIntroducing the New Colleague - ChatGPT