Optimizing and Benchmarking Open-Source LLMs
summary
Introduces a new metric Repetition-Aware Performance (RAP) and an efficient repetition-detection algorithm that classifies repeated tokens and measures a repetition ratio, then applies a cubic penalty to accuracy; reports the detector is ~20% faster than prior work. Benchmarks use WebQuestionsSP and Microsoft MARCO, sweeping the repetition penalty (RPP) from 1.0 to 1.3 in 0.02 steps. Swapping models in a fixed RAG pipeline shows rag chat template prompting cuts repetition by ~10× and can reduce repetition by 92% with only a 0.022 F1 drop.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 1:14Problem with open-source LLMs
- 3:01Repetition penalty parameter challenges
- 4:05Introducing Repetition Web Performance
- 5:52Benchmarking methodology
- 7:00Evaluation datasets and metrics
- 7:41Prompt engineering strategies
- 8:44Key findings from experiments
- 10:07Balancing performance and repetition
- 11:31Algorithm efficiency and RWP impact
- 12:17Key takeaways and conclusion
related talks
31mBeyond LLMs and Maturing RAG use cases
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
27mOptimizing RAG Systems in Practice: Challenges and Proven Solutions
50mKa-Ching! How Enterprises Measure LLM Performance and Cost
29mPutting the data science back into LLM evaluation
15mEvaluating RAG Model Performance Metrics, Bias, and Interpretability
52mReduce LLM calls with Vector Search Design Patterns
48mCrafting your own RAG system: Leveraging 30+ LLMs for enhanced performance