From model weights to API endpoint with TensorRT LLM
summary
TensorRT-LLM is shown as a production-ready toolkit that converts Hugging Face weights into GPU-specific TensorRT engines and runs them via Triton or a custom server; the workshop builds a Tiny Llama 1.1B engine on 4090s, tests FP8 quantization, and measures real numbers: engine drops from ~2 GB to ~1.2 GB after FP8, streaming latency around 118–180 ms time to first token, and throughput from ~5k to ~7k tokens/sec for tiny models (Mistral 7B example hits ~216 tps with 180 ms TTF); engines require matching GPUs and careful batching to avoid slow cold starts.
AI-written summary · about
chapters (11)
- 0:00Introduction to TensorRT
- 2:58What is TensorRT-LLM
- 10:26Challenges and performance
- 15:17Supported models and features
- 23:00TensorRT vs other frameworks
- 27:25Setting up the environment
- 38:04Building an engine
- 44:54Inference and optimization
- 56:09Automated deployment with Truss
- 1:03:58Benchmarking and scaling
- 1:17:20Live benchmarking workshop
related talks
12mDemo: Optimizing Gemma inference on NVIDIA GPUs with TensorRT-LLM
29mLeveraging open-source LLMs for production
01h 21mTLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs
30mOur Journey With TorchTitan
13mLightning Talk: From Hugging Face To Handheld: Scaling LLM Deployment W...
20mUnlocking Performance: Harnessing LLMs To Streamline GPU Kernel Development in...
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...