Sponsored Keynote: Lessons Learned in LLM Prompt Security
Talkat KubeCon + CloudNativeCon Europe 2025 · on Friday, 2025-04-04· at 09:55· Level 0 | ICC Auditorium·
summary
HAProxy load balancers run prompt-classification models at the edge to detect jailbreaks and PII but performance becomes the bottleneck: 150–200 ms per 500 tokens on a G6X large, scaled prompts hit 2,000 tokens and multiply latency, and throughput tops out at ~60 RPS (drops below 40 at concurrency 8). Optimized inference yields roughly +30% speed, token caching helps little for classification, and simple text filters fail on mangled words. Smaller purpose-built models are proposed as the next step.
AI-written summary · about
related talks
45mPrompt Hardening - Secure Code Generation Using AI
30mSecuring LLMs in Kubernetes: Best Practices
02h 01mPrompt Engineering and AI Red Teaming
26mOne guardrail is not enough: Defense-in-depth against LLM jailbreaks
01h 00mPrompt Failures and Latency Spikes: Observability for AI
46mIncreasing the Analysis Surface of Large Language Models
26mHow To Win Friends & Influence LLMs (with Prompt Engineering)
15mAI Trends, and Running LLMs in the Enterprise