Deploying ML solutions with low latency in Python
summary
Shows concrete steps and numbers for cutting inference latency: convert PyTorch/TensorFlow models to ONNX then optimize with TensorRT using FP16/INT8 calibration, kernel auto-tuning, layer fusion and serialized engines; reports RetinaFace ResNet-50 on GTX 1080 at ~81 FPS (FP32) vs ~190 FPS (INT8), RetinaFace MobileNet ~400 FPS, YOLOv5 large reaches real‑time, and a T4 server hits ~1,100 FPS while Jetson Nano face detection reaches ~95 FPS. Also covers building GStreamer pipelines and batching with DeepStream and optional Triton integration.
AI-written summary · about
chapters (12)
- 0:00Low-latency ML deployment intro
- 1:47Latency and throughput basics
- 2:48Quantization and pruning
- 3:52Knowledge distillation
- 4:13TensorRT framework
- 7:46TensorRT optimization
- 10:55Converting to TensorRT engines
- 12:57TensorRT deployment best practices
- 13:59DeepStream framework
- 15:23DeepStream pipeline steps
- 16:51DeepStream performance metrics
- 17:34Resources and closing
related talks
23mThe Race to the Bottom: Low Latency in the age of the Transformer
53mMachine Learning Platforms
23mOptimize your network inference time with OpenVINO
14mLightning Talk: Every Millisecond Counts: The Fine-tuning Journey of an Ultra-Eff...
16mLightning Talk: Efficient Inference at the Edge: Performance You Need at the Lowest...
36mAccelerating Machine Learning at Scale with
31mLow-Latency Solutions for Resource-Constrained Environments
45mExtracting real-time data at the edge with SageMaker & NVIDIA GPUs