How to choose the right instance type for ML inference
summary
Benchmarks show concrete tradeoffs: a ResNet50 on a G4 T4 runs TensorFlow at ~115 images/sec and ~17 ms, and after Nvidia TensorRT compilation jumps to ~1,600–1,800 images/sec with single‑digit ms latencies (ResNet50: 115→1,600+ img/s with TensorRT). A C4 CPU gives ~26 img/s at ~300 ms, while Amazon Elastic Inference on the same host hits ~160 img/s at ~50 ms (CPU 26 img/s 300 ms → EI 160 img/s 50 ms). Inf1/Inferentia requires compilation, reports neuron coverage ~99%, and often matches GPUs at lower cost. Tools: SageMaker, TensorRT, Elastic Inference, Inf1.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:15Inference performance affects customer experience
- 2:13Speed up inference with an accelerator
- 3:13Many options: Where do you start?
- 11:21Benefits of GPU-accelerated inference
- 12:34GPU key strength: Programmability GPUs are usually first to get support for novel models and operators You can write custom GPU accelerated model using NVIDIA CUDA
- 14:04Choosing the right G4 instance size
- 19:44Benefits of El accelerated inference
- 22:11AWS Inferentia: Custom silicon for ML inference
- 26:29Benefits of AWS Inferentia
- 28:28Instances and accelerators for ML inference
related talks
23mThe Race to the Bottom: Low Latency in the age of the Transformer
42mWhat's New with TensorFlow?
24mGenerative AI Chips
18mDeploying ML solutions with low latency in Python
30mGo, ONNX and AI in Production
43mNeural Networks: Fast GPU Inference with TensorRT
31mRealize price-performance advantage for AI
35mAccelerate your ML innovation with AWS ML infrastructure services