Evaluating LLMs under Concept Drift in Production
summary
Claims a new metric called the drift responsiveness score that flags concept drift early by taking the ratio of model performance on a recent “drift” set to a baseline set. Uses SBERT embeddings, PCA, and divergence tests (PSI and MMD) plus three LLMs (GPT, LLaMA, a smaller LM) on e‑commerce reviews spanning 2015–2024. Finds the small LM drops to DRS 0.89 while others stay high, showing DRS detects semantic shift that accuracy/ROUGE miss.
AI-written summary · about
chapters (21)
- 0:00Concept drift in LLMs
- 1:42Talk overview
- 2:44Semantic drift example
- 4:26Data vs concept drift
- 6:33Detecting drift in LLMs
- 9:19Drift detection limits
- 10:23Drift responsiveness score
- 14:58Simulation with Siddharth
- 15:42Exploratory data analysis
- 16:42Temporal data splitting
- 17:47Creating embeddings
- 18:34Evaluating drift performance
- 19:16DRS insights
- 20:22E-commerce segmentation
- 22:09Drift detection metrics
- 23:31PCA for semantic shifts
- 25:15LLM performance vs DRS
- 27:46DRS by product category
- 29:08Key takeaways
- 30:57DRS next steps
- 32:01Conclusion
related talks
39mHallucination-Free LLMs: Strategies for Monitoring and Mitigation
30mLLM-Based Recommendation Systems
41mControlling Dependency on Third‑Party Large Language Models
22mPractical Strategies for Implementing Domain-Specific LLMs
44mProduction-Grade LLMs & AI Agents
35mEvaluating LLM systems: the nuts and bolts
20mHow to Survive the Vortex of LLM Change
41mHow continuous testing keeps your LLM on track