From Supercomputing to Serving: A Case Study Delivering Cloud Native Foundation Models
Talkat Cloud Native + Kubernetes AI Day NA 2024 · on Tuesday, 2024-11-12· at 11:50· Salt Palace | Level 1 | Grand Ballroom A·
summary
A small team scales from one cloud to clusters across four cloud providers in under six months by building cross‑cloud plumbing that bursts stateless inference to remote GPUs using admirality proxy pods, and by running multi‑cluster scheduling with Kueue. A custom deployer based on Argo Workflows and Hera stitches seven teams’ steps into repeatable multi‑cloud deployments and actually works in production. Challenges include high GPU failure rates, inconsistent provider diagnostics, tight hardware–software coupling, and tricky quota and preemption behavior.
AI-written summary · about
chapters (8)
- 0:00Cloud-native AI infrastructure
- 4:34GPU availability limits
- 8:40GPU training cluster failures
- 12:53Tightly coupled AI software
- 16:06Post-training deployment challenges
- 19:59Accelerator allocation in lifecycles
- 22:50Q scheduler for multicloud GPUs
- 23:13Q&A
related talks
47mCloud Native and Sustainable, Reproducible Scientific Computing
36mAn almost perfect match: cloud native and scientific computing
19mKeynote: Building Bridges: Cloud Native and High Performance Computing
18mKeynote: Mind the Gap: Bridging Cloud Native Innovation with Real-World Use
24mBeyond Tool Sprawl: An Event-Driven Approach to Dynamic Multi-Cloud Operations
27mGen-AI at Scale: Simplifying Orchestration of Healthcare Applications Across Multi-Cluster Environme
10mLightning Talk: Does Cloud Elasticity Pay off for HPC Workloads?
35mCase Study: AI-as-a-Service on Kubernetes at Scale and In Production