Confidently iterate on GenAI applications with Weave
summary
Weave is a toolkit that you add with one line of code to capture prompts, model outputs, tracebacks and the wrapper code so debugging shows the exact failure mode. Demos include a document-extraction bug where the LLM correctly returns multiple share classes, and an internal Slackbot evaluated on 146 threads that improves from 15→32 answerable questions by upgrading models and changing retrieval to 4 documents. The system automatically captures code, params and per-example evidence so experiments and evaluation are reproducible.
AI-written summary · about
chapters (9)
- 0:00GenAI application tools
- 1:14LLMs in production challenges
- 3:01Building intuition via experiments
- 5:51Debugging LLM apps with Weave
- 8:42Slackbot case study
- 10:50Trustworthy LLM responses
- 13:22Evaluating LLM performance
- 15:51Efficient model iteration
- 18:57Summary and Weave future
related talks
15mDemystifying generative AI
40mDavid Fogl: Leveraging Generative AI for Enhanced Software Testing
24mHow to Test AI Features Before Your Users Do
45mPanel Discussion on AI for Code Generation
28mAPIs and GenAI : Your Perfect Combo with LLM-Powered Frameworks.
48mMastering the AI Shift: Adaptive Strategies for Software Evolution
12mAI-generated recap of Devoxx Belgium 2024 opening keynote : Genesis
23mBeyond Tests What to Verify in AI Generated Code