User Namespaces in Production: Enabling Root in Containers With RWX
summary
Kubernetes user namespaces are enabled across a production ML platform (2,000+ GPUs, 400 nodes, 100+ TB of data) by skipping failing idmapped mounts for NFS and making server-side IDs deterministic; container creation errors from mount_setid are avoided with an NRI plugin that removes idmap fields for NFS mounts, and exports are configured to UID/GID squash to 65534 so multiple pods see the same mapped IDs. The approach lets RWX volumes work without migrating storage, less than 10% of workloads break (git/chown/checkpoint tools) and require remediation.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:04Platform security challenges
- 3:15Container escape risks
- 6:35User namespaces basics
- 9:07Storage requirements
- 12:03ID mapping mounts
- 13:52RWX access limitations
- 15:16NFS RWX barrier
- 17:24Runtime workaround for NFS
- 19:49Non-deterministic ID mapping
- 21:54Safety of workaround
- 23:40Future work
- 25:07Key takeaways
- 26:35Q&A
related talks
21mMaking Kubernetes Safer with User Namespaces
40mUser Namespaces in Kubernetes: Security and Flexibility - Pick Both
30mThe Route To Rootless Containers
22mRun As “Root”, Not Root: User Namespaces In K8s
25mIsolate the Users! Supporting User Namespaces in K8s for Increased Security
32mRoot Without Risk: A Decade-Long Quest for True Container Isolation
35mThe Reality of Rootless Containers
42mWhat's New in the User Namespace