Sponsored Session: Accelerating GenAI Inference: From AWS Deep Learning...
summary
AWS Deep Learning Containers plus VLM and PyTorch power a multi‑node inference cluster that shards models across GPUs and Trainium using Neuron and EFA over libfabric, targeting 3 million tokens per minute throughput and sub‑second streaming latency. Architecture uses a leader/follower broadcast with continuous batching, hybrid parallelism (context‑parallel prefill, data‑parallel decode), and KV offload options to remove model‑size limits; production scale claim is 80,000 Trainium chips and 30–40% better price‑performance.
AI-written summary · about
chapters (14)
- 0:00AWS GenAI inference innovations
- 2:30LLM inference infrastructure shifts
- 4:19Decoupling prefill and decoding
- 5:45Mixture of Experts challenges
- 7:08Enabling networking with EFA
- 8:58VLM framework and DL containers
- 10:24Amazon Rufus and Trainium scaling
- 12:31Rufus key challenges
- 15:02Multi-node inference cluster
- 18:39Hybrid parallelism strategies
- 20:24Containerized solution flexibility
- 21:47Reliability with telemetry
- 22:29Rufus infrastructure goals
- 23:53Innovate with AWS building blocks
related talks
24mGenerative AI Chips
22mGenAI Training In Production: Software, Hardware & Network Considerations
14mAccelerating AI inference workloads
54mDeep learning on AWS with NVIDIA: From training to deployment
19mServerless inferencing: an infrastructure point of view
13mTailor-Made Inference at Scale
52mSimplifying the adoption of generative AI for enterprises
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…