Shift-Left for LLMs: Securing the AI Model Supply Chain
summary
A Kubeflow pipeline fetches a 0.5B model from Hugging Face, stores artifacts in MinIO, applies Sigstore and Cosign signing, runs NVIDIA Garak for behavioral/vulnerability scans and emits JSON risk scores, injects NVIDIA NeMo guardrails pre/post prompt, and deploys via KServe with an Envoy gateway enforcing token rate limits (10-request demo) and Istio-like pod network controls. The end-to-end demo runs ~30 minutes across five–six stages, produces a Garak report, and validates inference (Chinese model output) and signing checks before serving.
AI-written summary · about
chapters (11)
- 0:00LLM security challenges
- 3:36Attack areas in LLMs
- 8:56Shift-left for LLMs
- 12:05Model acquisition & verification
- 15:59Demo setup & pipeline
- 19:14Executing security pipeline
- 23:28Pipeline results walkthrough
- 28:22Model integrity validation
- 31:17Deployed model testing
- 33:37Continuous security & red teaming
- 34:48Q&A
related talks
25mSecuring AI Pipelines: Real-World Attacks on Kubernetes-Based AI Infrastructure
28mNavigating LLM deployment: Tips, tricks, and techniques
32mFrom Edge To Insight: Cybersecurity Intelligence With Kubeflow...
33mScalable LLM Inference on Kubernetes With NVIDIA NIMS, LangChain, Milvus and Flu...
30mOwned by Statistics: Using Kubeflow to Defend vs Attacks on Your ML Models
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
40mMLSecOps with Automated Online and Offline ML Model Evaluations on Kubernetes
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine