Building High-Quality & Trustworthy FMware
summary
Practical roadmap for production: start with medium or smaller models rather than a 100B+ default because latency, cost and edge limits matter; use generic benchmarks (MMLU, GLUE, MT-Bench, Chatbot Arena) then build a domain-specific benchmark since many benchmarks (HumanEval) are saturated or leaked into training. Tune progressively: prompt → PEFT adapters → full finetune → RL with verifiable rewards; watch data quality, avoid pure synthetic collapse. Add retrieval guardrails, Nemo Guardrails or open-source guardrail AI, and layered input/output filters plus monitoring for semantic observability.
AI-written summary · about
chapters (15)
- 0:00Trustworthy foundation model software
- 2:08Selecting and optimizing models
- 5:20Fine-tuning and alignment
- 7:25Data quality and synthetic risks
- 7:48Prompt engineering challenges
- 9:14Debugging with explainability
- 10:39Context engineering pipelines
- 12:22Evaluating model applications
- 14:50Limitations of LM judge
- 16:14Guardrails against injection
- 18:45Constitution AI safety
- 19:51Trust in multi-agent systems
- 21:17Balancing user trust and limits
- 22:01Monitoring AI systems
- 23:06Balancing quality and trust
related talks
41mFrom Lab To Life: Practical AI System Evaluation
38mBuilding AI responsibly with a GRC strategy
19mBuilding AI Products That Actually Work
28mNavigating LLM deployment: Tips, tricks, and techniques
46mEnhancing Agent Reliability Through Evaluation-Driven Development
20mEverything I Learned Training Frontier Small Models
15mPractical tactics to build reliable AI apps
39mForward Focus: Perspectives on AI, Hype, and Security