Model innovation & evaluation with model partners Q&A
summary
Discussion recommends using LMs to judge other LMs and insists on production-grade observability to collect traces and evaluations. RAG is decomposed into retriever metrics and generator faithfulness with an end-to-end KILT benchmark example. Complexity from LinkedChains and LlamaIndex means one-line demos hide hundreds of calls, making debugging the last 20% hard. Practical tools mentioned include Arize Phoenix open-source tracing/eval, MultiHop/MultiQuery RAG techniques, and measuring each component separately before deployment.
AI-written summary · about
chapters (9)
- 0:00Model innovation intro
- 0:532024 language model differences
- 1:56Evaluating language model applications
- 2:59Enterprise RAG focus
- 4:03Observability and AI eval
- 5:11RAG eval best practices
- 6:37AI observability shifts
- 7:19Demo to production challenges
- 8:24Final thoughts
related talks
34mEvaluating LLM Frameworks
38mMaking LLMs reliable - A practical framework
01h 28mMastering RAG Evaluation with MLflow & Open Source LLMs
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
31mBeyond LLMs and Maturing RAG use cases
15mEvaluating RAG Model Performance Metrics, Bias, and Interpretability
42mAdvancements in Evaluating Large Language Model Applications
35mEvaluating LLM systems: the nuts and bolts