Shared Yet Isolated at Scale: Building Multi-Tenant Inference Platform on…
summary
SoftBank runs over 10,000 GPUs (A100, H100, DB200) and builds a vcluster-based platform that gives teams either vcluster virtual clusters with full cluster-admin scope, a serverless shared LLM endpoint, or dedicated GPU nodes. Engineers implement a GPU bin-packing scheduler (mostAllocated with high GPU weight) to eliminate fragmentation and fit larger 6+ GPU jobs, and use cache salt per-tenant to avoid prefix-cache leakage. Benchmarks show prefix-aware routing dramatically raises cache hit rate and cuts time-to-first-token.
AI-written summary · about
chapters (11)
- 0:00Introduction to the platform
- 1:11AI infrastructure background
- 2:04Challenges of multi-tenancy
- 5:24Virtual cluster solution
- 7:12Platform architecture and CLI
- 10:19GPU scheduling optimization
- 11:50Multi-tenancy models
- 13:50Tenant access and observability
- 18:27Secure prefix caching
- 21:55Intelligent routing and scaling
- 26:05Conclusion and future works
related talks
21mExperience Operating Large GPU Clusters at Organizational Scale
09mLightning Talk: Bringing Multi-Tenancy to GPU-Based Workloads
01h 15mTutorial: Build, Operate, and Use a Multi-Tenant AI Cluster Based Entirely on Open Source
28mGPU Reservations: Maximizing Utilization and Fairness Across Teams
28mGPU sharing done right: Secrets, security, and scaling with Vault and vCluster
38mStop Allocating GPUs, Start Delivering Intelligence: An Enterprise...
07mYour GPU delivery is here! Now what? A 5 minute guide to make your CTO happy
12mVertical Mobility: Inference from MVP to Trillion Parameters