Monitoring GenAI Applications
summary
Monocle ships a spec plus Python and TypeScript SDKs and a VS Code extension that generate OpenTelemetry-compatible traces with little or no code change, capturing prompts, responses, token counts, retrieval spans (example: Chroma vector store), and model calls across hybrid deployments (OpenAI, Azure OpenAI, Triton). Demos show a RAG chatbot and a Teams bot instrumented via a wrapper or minor imports, traces exported to S3/Azure or Okahu AI, and scopes that stitch per-conversation traces for cost, latency and error analysis.
AI-written summary · about
chapters (12)
- 0:00GenAI monitoring challenges
- 3:11GenAI app complexity
- 4:00Simple RAG chatbot demo
- 5:56Team-based chatbot example
- 7:45Why GenAI monitoring differs
- 9:10Monle for GenAI instrumentation
- 10:14Instrumenting simple chatbot
- 13:07Instrumenting Teams chatbot
- 16:07Monle architecture benefits
- 19:12Scopes and analytics
- 21:17Oahu AI production demo
- 22:45Call to action
related talks
22mMonitoring GenAI Applications
39mMonitoring Agentic Applications
40mOptimizing AI Applications for Production with Observability
31mDeep Dive To AI Agent Observability
01h 02mObserving AI Applications with OpenLit and OpenTelemetry
27mKeynote: Tracing the Agent's Mind: Extending OpenTelemetry for Deep...
38mAI Monitoring Mastery: Optimize Performance, Ensure Compliance, and Cut Costs
25mMonitoring AI with AI