Task Fidelity Scaling Laws
summary
Experiment uses same model same compute same number of tasks and fine tunes on TerminalBench-style agentic tasks, yielding concrete numbers: low-quality tasks give ~1% vs 6% improvement and a 5x uplift from task quality alone. Accepted tasks average twice as many tool calls, lower pass rates, and more output tokens, with cleaner failure modes tied to real logical gaps; rejected tasks fail from spec-test mismatches or missing implicit context.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:39Core thesis
- 3:22Task quality criteria
- 5:07Accepted vs rejected tasks
- 8:44Task quality impact
- 10:29Expert-in-the-loop data
- 11:54Q&A
- 13:40Handling under-specified tasks
- 16:31Future challenges
- 17:54Inter-annotator agreement
- 20:01Closing remarks
related talks
32mReproducibility in Embedding Benchmarks
21mBeat GPT 4 and Build the Best AI For You: Mobilizing First-Party Data with Fine-Tuning
39mThe practice of acquiring good labels
53mUnleashing the Black Box: Testing Machine Learning Systems
35mCreating Your Own LLM Tuning Platform with Open Source Technologies
25mReproducibility in Embedding Benchmarks
15mSuccessfully Scaling AI and the Role of Trust
29mLocal LLM fine-tuning: a practical example