Accelerating AI inference workloads
summary
JetStream delivers an open‑source, XLA‑optimized inference engine for large language models with techniques like continuous batching and quantization to cut latency and cost; MaxDiffusion and MaxText provide high‑performance diffusion and LLM reference implementations for Cloud TPUs and Cloud GPUs (H100 / upcoming HGX B200). JAX and PyTorch ports exist on GitHub, and deployment uses GKE autoscaling (including scale‑to‑zero) to manage multi‑accelerator fleets and control costs.
AI-written summary · about
chapters (7)
- 0:00Meet Alex
- 2:52Balancing cost and efficiency
- 5:51TPU vs GPU for AI models
- 8:21Getting started with Google Cloud TPUs and GPUs
- 10:05Common challenges when using inference optimization
- 12:10Available resources for AI inference workloads
- 13:13Wrap up
related talks
25mSponsored Session: Accelerating GenAI Inference: From AWS Deep Learning...
36mAccelerating High-Performance Machine Learning at Scale i...
55mLarge Scale Distributed AI on the JVM
12mAI-powered infrastructure:
04mState-of-the-art Machine Learning research with Google tools
31mHigh performance machine learning with JAX
05mAccelerate AI development from cloud to client
36mApplied AI on the edge