Testing LLM-Powered Applications
summary
Case study of Microsoft's Tay shows a chatbot goes off the rails in 42 hours, and a resume-screener demo reveals prompt injection via white text hidden in PDFs that forces the model to pick a particular candidate. Testing demands factual/unit tests, property-based checks, adversarial prompts, and BLEU and ROUGE comparisons plus judge-LLM auto-eval with discrete star scoring. Recommended automation and tooling include DeepEval Ragas DeepChecks Trens to collect prompts run cycles and monitor non-determinism and hallucinations.
AI-written summary · about
chapters (12)
- 0:00Introduction and Welcome
- 0:31Case Study: Microsoft's Tay Bot
- 2:08The Risks of LLMs in Real-World Applications
- 2:39Testing LLM-Powered Applications
- 3:29Security Concerns and Prompt Injection
- 4:40Non-Determinism and Inaccuracy in LLMs
- 6:25Building a Robust Test System
- 8:04Types of Testing for LLMs
- 9:41Metrics for Evaluating LLMs
- 13:50Adversarial Testing and Auto Evaluation
- 16:54Open Source Tools for LLM Testing
- 18:37Conclusion and Final Thoughts
related talks
39mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
47mCan You Trust Your ChatBot: Techniques for Testing LLM Responses
42mRobust Testing Strategies for AI-Driven Applications
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
10mMastering LLM Chatbot Testing: Metrics, Methods and Mistakes to Avoid
13mNew Challenges in Software Quality Brought by GenAI and LLMs
01h 04mWhat are we thinking in the age of AI?
24mHow to Keep Your LLM Chatbots Real - A Metrics Survival Guide