Neural Networks: Fast GPU Inference with TensorRT
summary
Describes accelerating neural-network inference on GPUs using NVIDIA TensorRT: explains that training frameworks trade performance for flexibility, so deployed models need optimization, and demonstrates how TensorRT can perform fast inference of trained networks on GPU hardware.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:04Convolutional networks
- 6:04Training to inference
- 7:49What is TensorRT
- 11:44TensorRT optimizations
- 16:20Demo: model training
- 21:15Freezing model graph
- 24:30Inference in TensorFlow
- 26:15Inference in TensorRT
- 32:18Conclusion
- 34:24Q&A
related talks
51mTips & Tricks for Fast Neural Net Inference in Production
18mDeploying ML solutions with low latency in Python
37mDeep Learning Neural Network Acceleration at the Edge
30mUnderstanding GPUs
32mHow to choose the right instance type for ML inference
13mLightning Talk: Accelerated Inference in PyTorch 2.X with Torch...
23mThe Race to the Bottom: Low Latency in the age of the Transformer
32mScaling Inference Deployments with NVIDIA Triton Inference Server and Ray Serve