Prompt Variation as a Diagnostic Tool
summary
Claim: small prompt changes collapse accuracy — examples show 95% to 75% drop on rephrasings and Microsoft finds a 40% shift for GPT‑3.5 Turbo between markdown and JSON templates. Tools and measures include lexical, structural, order and reasoning variations, mixture‑of‑formats and the spread metric; mixture‑of‑formats reduces spread, JSON sometimes raises accuracy, and prompt masking reveals dataset memorization. Evaluation pipelines incorporate these diagnostics and they do expose contamination and brittle pattern‑matching.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:06Prompt sensitivity in LLMs
- 3:54Prompt formatting impact
- 5:21Lexical and structural variations
- 8:10Order sensitivity and reasoning
- 11:42Mixture of formats
- 15:33Stability and generalization
- 16:16Detecting data contamination
- 19:25Contamination-free benchmarks
- 22:36Evaluation pipeline integration
- 23:29Q&A
related talks
38mPrompt Engineering ⚙️ - Addressing the sensitivity of Large language models
01h 08mIn Prompts We Trust
13mPrompt Engineering Simplified
01h 04mIn Prompts We Trust
43mAdvanced Prompt Engineering and Beyond
29mIs Prompt Engineering Dead? How Auto-Optimization is Changing the Game
16mNavigating the Uncharted: Ensuring Prompt Quality in the Age of Language Models
18mTechniques for Enhanced Model Performance