Beyond Monolithic AI: Cloud Native Patterns for Dynami...
summary
A concrete architecture for an AI Gateway that uses semantic routing (a small intent model plus rule signals) and a VLLM-based serving layer to pick models per request, halving latency and cutting token cost toward ~100% for trivial queries. A dynamic tool retrieval / tool-RAG limits cached tools to a handful, tripling tool-call accuracy in noisy enterprise catalogs. Deployed as an Envoy gateway with a semantic-route CRD (policy-as-code) and observability via Jaeger, Grafana and OpenTelemetry; demo shows guardrails blocking PII and routing heavier queries to reasoning models with judge scores.
AI-written summary · about
chapters (8)
- 0:00Cloud-native AI agents intro
- 3:12AI agent deployment challenges
- 7:23Why cloud-native fits AI
- 9:28Agentic systems architecture
- 15:42Behavioral science in planning
- 18:08CNCF tools for AI
- 21:41Semantic routing demo
- 27:41Key takeaways
related talks
01h 02mArchitecture Patterns for AI-Powered Applications
23mThe State of GenAI & ML in the Cloud Native Ecosystem
42mDefense against the dark arts of AI: The playbook for a sovereign model-as-a-service platform
37mEvolve, Adapt, and Adopt: Envoy in the AI Era
48mHow to Make Smart Architecture Decisions when Building Gen AI Apps
29mNavigating Innovation with Open Hybrid Cloud and Openshift AI
42mEffortlessly Build, Run, Secure, and Manage Traffic for a Generative AI Application From...
08mArchitecting Transparent AI Systems for Cloud-Native Environments