Going production scale with Gen AI - Patterns for your API
summary
A production pipeline is built with an API fronting Azure OpenAI and OpenAI ChatGPT, storing every message, token counts and timestamps in a message analytics DB (Cosmos or Postgres), applying rate limiting and content filters, and validating JSON outputs and session IDs. A live demo runs ~2,800 messages and records 584 content-filter/retry events; references and quotas prevent exfiltration. Prompt-injection tests, monitoring of latency/failures, and an open-source code release are in place.
AI-written summary · about
chapters (15)
- 0:00Introduction and setup
- 2:22Core architecture patterns
- 7:06State management
- 9:24Limiting resources and cost
- 13:08Safety and content controls
- 15:51Analytics and session tracking
- 19:30Feedback and validation
- 24:04Short-circuiting and filtering
- 30:40Structured data handling
- 33:59Advanced security patterns
- 38:24RAG and fine-tuning
- 41:25Resilience and scaling
- 44:23Monitoring and observability
- 48:56Testing methodologies
- 52:07Security conclusion
related talks
48mTake an Azure OpenAI chat application from PoC to enterprise-ready
24mInfuse Generative AI in your apps using
40mOptimizing AI Applications for Production with Observability
30mClean and Controlled AI with API Management
40mBuilding secure and efficient AI solutions in Azure with Partners
24mScalable GenAI in Production
52mDesigning Production-Ready Agentic Systems: Lessons from the Field
01h 26mAutomation of the Automation using GenAI