Inside Microsoft AI innovation with Mark Russinovich
summary
Azure reveals that training scale drives every design choice: a Llama‑370B needs roughly 6.4 million H100 GPU‑hours (one GPU would take ~730 years, a slice ~27 days), GB200 draws ~1.2 kW per GPU, and a server ships ~3.2 Tb/s out. Microsoft builds Maya liquid cooling and Maya liquid‑cooled racks to cut power/water, deploys Project Forge global scheduler to lift utilization from ~50–60% to ~80–90%, and uses a storage accelerator that loads a 270 GB model in 12 min from blob storage but under half that from the cluster cache. Confidential H100 GPUs attest and protect models and prompts end‑to‑end.
AI-written summary · about
chapters (39)
- 0:00Microsoft AI innovation intro
- 1:16AI stack infrastructure overview
- 2:00AI model sizes growth
- 3:03AI supercomputers for OpenAI
- 4:05Llama 3 training times
- 5:28Memory for inference and training
- 6:58GPU high bandwidth memory evolution
- 8:04Diverse AI hardware in Azure
- 8:48Power consumption challenges
- 9:56Liquid cooling for GPUs
- 12:43Sustainable data center design
- 13:03AI workload power patterns
- 14:23Power over-subscription efficiency
- 15:26Networking for AI training
- 16:57Azure InfiniBand network
- 18:49Data center cabling innovations
- 19:51Storage for AI model distribution
- 21:59Storage accelerator demo
- 22:19Project Forge resource management
- 23:02Global GPU pooling efficiency
- 24:53AI workload reliability automation
- 27:42Language model inference anatomy
- 28:48Prompt vs token generation
- 30:15Project Flywheel serverless GPU
- 31:41Project Flywheel efficiency demo
- 33:49LoRA fine-tuning
- 36:18MultiLoRA concurrent serving
- 37:21MultiLoRA demo
- 39:13Confidential computing evolution
- 41:06Trusted execution environments
- 43:11Confidential GPUs for AI
- 44:14Confidential computing scenarios
- 46:02Confidential RAG demo
- 48:31Future AI innovations conclusion
- 48:53Copilot personal AI research
- 51:03Model unlearning for privacy
- 52:28Forgetting profanity in models
- 54:15AI in software development today
- 55:19Closing remarks
related talks
46mInside Azure innovations with Mark Russinovich
21mDesigning Scalable Networks for Large AI Clusters: Challenges and Key Insights
13mFrom Silicon to Scale: Training at Scale
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud
24mManipulating Data in an Artificial Intelligence World
34mScaling AI Like a Pro-PepsiCo’s LLM Deployment Strategy on K...
28mComputing Large Foundational Models
09mDemocratizing Access to GPUs for Faster MLAI Innovation with Generative AI