High-Scale Networking for ML Workloads With Cilium
summary
Production Cilium on over 10,000 bare‑metal nodes (clusters up to 1,000) runs eBPF dataplane for ML jobs processing 4–5M daily (peaks 11M). A policy map hit 110,000 errors/s when Armada jobID labels created tens of thousands of identities; fixing identity selection (exclude Armada labels) collapses map pressure. Conntrack GC spikes (10–20s) cause endpoint delete timeouts; capping GC interval to 10 minutes via Helm removes the tail. A deployment with GoGC disabled produces 1 TB memory in Cilium pods.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 3:07Deployment overview
- 7:04Map pressure failures
- 15:30Connection tracking GC
- 22:13Memory usage debugging
- 23:18Q&A
related talks
25mMore Churn No Problem: Lessons Learned Running Cilium in Production
20mTackling Container Networking at Scale
16mCilium Wars: Scaling Up Your Network with Jedi-like Cilium Handling Techniques - Yash Tandon
36mCilium: Welcome, Vision and Updates
32mSwitching your CNI — Simone Sciarrati & Federico Hernandez
05mLightning Talk: Unveiling 5 Essential Insights from Cilium Implementation in a...
28mA Decade of Cilium Around the World
28mCloud native for governmental workloads