Policy-as-Code for LLM Inference: Cost & Security Guardrails
summary
Kyverno validating and mutating policies enforce GPU and cost guardrails at admission time for Kubernetes LLM inference, rejecting deployments that exceed token or GPU limits and injecting cost annotations for chargeback. Examples: require nvidia.com/gpu limits, block H100 in dev, inject a GPT-cache sidecar for semantic caching, and compute pricing from A100 ~ $3.5/hour (deployments can reach $10k–$50k/month). Auditing with Prometheus/VictoriaMetrics + Grafana enables tuning before enforcement and yields 40–60% cache hit savings.
AI-written summary · about
chapters (12)
- 0:00Policy-driven LLM inference
- 1:01LLM training vs inference
- 3:08Why LLM inference costs
- 4:54LLM inference on Kubernetes
- 5:38Traditional controls fail for GPU costs
- 7:04Cost control with Kyverno
- 8:31GPU resource limit policies
- 9:56Block expensive hardware in dev
- 11:00Semantic caching with Kyverno
- 12:49Cost guardrails implementation
- 13:55Key takeaways
- 14:17Q&A
related talks
42mStop Treating LLMs Like REST APIs
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
30mSecuring AI Workloads: Building Zero-Trust Architecture for LLM Applications
38mStop Allocating GPUs, Start Delivering Intelligence: An Enterprise...
24mTaming the Wild West of Research Computing: How Policies Saved Us a Thousand...
25mAgentic Zero Trust: at Rest, in Transit, and at Runtime
46mGuarding the LLM Galaxy: Security, Privacy, and Guardrails in the...
27mWho's Using That GPU? Identity-Aware Access Control for Kubernetes GPU…