Code-First LLMOps from prototype to production with GenAI tools
summary
Prototype becomes deployable: a code-first pipeline uses Prompty files checked into Git, generates LangChain/Semantic Kernel code, and deploys a multi‑agent creative writer on AKS with azd up in ~14 minutes. PromptFlow tracing captures prompts, tool calls and vectors; evaluators score relevance/fluency/groundedness. A cutoff bug is fixed by changing requested word length and raising max_tokens from 512 to 1,200, after which CI GitHub Actions batch evals move scores from low to near‑5s. Production dashboards show token usage, latency, and recommend sampling to cut evaluator costs.
AI-written summary · about
chapters (12)
- 0:00LLM development journey
- 3:06Introducing Prompty
- 5:51Developing with Prompty
- 8:23Deploying with AZD
- 10:44Agentic AI application
- 13:21Debugging and tracing
- 18:41Information retrieval
- 24:26Evaluating application
- 27:35Fixing bugs and CICD
- 29:09Automated testing
- 31:53Production monitoring
- 35:19Key takeaways
related talks
42mCode First LLMOps With Azure AI & Prompty
43mBuild and Maintain your Company Copilot with Azure ML and GPT-4 Q&A
33mOperationalizing GenAI Applications with DevOps Practices
41mImproving DevOps workflows with GenAI
37mAI-Driven Developer Productivity at Scale
36mAgentic CI/CD: from Pull Request to Production without the Paper Cuts
01h 08mIn Prompts We Trust
26mMastering Large Language Models: Prompt Engineering for Precision & Performance