Securing AI Workloads: Building Zero-Trust Architecture for LLM Applications
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 15:15· Level 0 | ICC Auditorium·
summary
Claims a practical zero‑trust stack for LLMs with concrete tools and attacks: apply AI gateway enforce ABAC/OIDC via Keycloak and OPA, use Guardrail AI, LLM‑guard, Vigil, Rebuff and Nvidia Garak for prompt‑injection and runtime scanning, sign model artifacts and use OCI registries, Kubernetes service mesh and container sandboxing for isolation. Cites real findings (CVE in Nvidia toolkit, ~12,000 exposed API keys, malicious “dips” packages) and demos a working kubectl MCP server prototype built days before the talk that issues cluster commands via LLM.
AI-written summary · about
chapters (16)
- 0:00Securing AI workloads
- 1:07LLM deployment methods
- 2:13LLM security incidents
- 4:48LLM request flow & attack surfaces
- 10:02AI gateways: roles & components
- 14:19Model inferencing risks
- 16:05Zero-trust for LLMs
- 18:12OWASP LLM risks
- 19:38Prompt injection threats
- 21:06Runtime security & API flaws
- 22:29Supply chain risks
- 23:54PII protection in LLM outputs
- 24:59AI gateway frameworks
- 26:05Zero-trust layers
- 27:51CNCF for zero-trust LLMs
- 28:11MCP server demo
related talks
19mSecuring AI Pipelines from Development to Production
22mBuilding Secure AI Platform Infrastructure: A Zero Trust Framework
29mBeyond the Hype: Building Trustworthy and Reliable LLM Applications with Guardrails
25mDe-risking Deployments in an AI World
40mIn AI We Trust? Securing the Future, One Agent at a Time
01h 01mIntroduction to AI Security
06mData Privacy Concerns and Solutions When using LLMs in Production
26mSponsored Session: Navigating Open Source AI with Guardrails