Prompt Engineering ⚙️ - Addressing the sensitivity of Large language models
summary
Mantis runs systematic comparisons of different prompts across over 6 models (GPTJ 6B, GPT‑3 175B, Bloom, Jurassic‑1, OPT, Cohere, Goose AI and others) on 5 NLP tasks (classification, QA, NER, similarity, summarization). Experiments use a 100‑sample per dataset baseline, show prompt changes giving 2x–3x score swings, that an ~11B model can approach a ~150B model with the right prompt, that instruction‑fine‑tuned models often outperform, and that prompt gains cut inference cost substantially.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:55Talk overview
- 2:37Large language models
- 4:04Ways to interact with large language models
- 8:02Prompt engineering
- 10:11Effective prompt guidelines
- 14:07Evaluation setup
- 16:38Key takeaways
- 17:43Prompt quality impact
- 18:25Complex prompts as a double-edged sword
- 20:07More shots do not always mean better performance
- 21:34Instruction fine-tuning
- 23:20Model cost factors
- 25:30Final thoughts
- 27:39Summary
- 28:02Q&A
related talks
02h 01mPrompt Engineering and AI Red Teaming
26mMastering Large Language Models: Prompt Engineering for Precision & Performance
13mPrompt Engineering Simplified
16mPrompt Engineering for Testers: Types and Applications
01h 56mPrompt Engineering: The Path to Effective Work with ChatGPT
18mTechniques for Enhanced Model Performance
30mIs Prompt Engineering Dead? How Auto-Optimization is Changing the Game
24mBetter Prompting with Pseudocode